Crawling the infinite web
Year: 2007
Type: article
Abstract: A large amount of the publicly available Web pages is generated dynamically upon request, and contain links to other dynamically generated pages. Many Web sites that are built with dynamic pages can create arbitrarily many pages. This poses a problem for the crawlers of Web search engines, as the network and storage resources required for indexing Web pages are neither infinite nor free. In this article, several probabilistic models for user browsing in "infinite" Web sites are proposed and studied.
Source: Journal of Web Engineering
Authors: Ricardo Baeza‐Yates, Carlos Castillo
Institutions: University of Chile, Universitat Pompeu Fabra
Language: English
Topic: Web Data Mining and Analysis
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W182781187
