Work

Crawling the infinite web

Year: 2007
Type: article
Abstract: A large amount of the publicly available Web pages is generated dynamically upon request, and contain links to other dynamically generated pages. Many Web sites that are built with dynamic pages can create arbitrarily many pages. This poses a problem for the crawlers of Web search engines, as the network and storage resources required for indexing Web pages are neither infinite nor free. In this article, several probabilistic models for user browsing in "infinite" Web sites are proposed and studied.
Language: English
Open Access status: diamond
OpenAlex ID: W182781187
FWCI: 5.236
Cites: 29
Cited by: 23
Related to: 10