An Efficient Algorithm for Webpage Change Detection For Network Load Reduction

Abstract: As the size of the web continues to grow, searching it for useful information has become increasingly difficult. Sufficient of current Internet traffic and bandwidth consumption are due to the web crawlers that retrieve pages for indexing by the different search engines. These crawlers also cause load on the remote server by using its CPU cycles and memory.
Moreover, due to the dynamic nature of the web, it becomes very difficult for a search engine to provide fresh information to the user. The web pages are uploaded and updated very frequently. There is a frequent change in the content of the web page hence it become necessary to develop an efficient system which could detect these changes efficiently and in the minimum browsing time. The mobile crawlers filter out pages that are not modified since the last crawl before sending them to the search engine for indexing purpose and to achieve this the old web page is compared with the new web page.The web page change detection system can be implemented by using various Tools or Algorithms.In this paper a new algorithm is presented to detect content level changes. It uses the text Code to detect content level changes by using the chararacter digit position in the word.

Keywords: Mobile crawler, Network load, Pagechange, Search engine.