Keyword

web crawling

Description: A method of systematically discovering and retrieving web pages by following links or other rules to access online resources.

Primary topic: Web Data Mining and Analysis (40%)

Wikidata ID: wiki/Q61466324

OpenAlex ID: keywords/web-crawling

The anatomy of a large-scale hypertextual Web search engine

1998 · Sergey Brin, Lawrence Page · Computer Networks and ISDN Systems · 15,991

Graph structure in the Web

2000 · Andrei Broder, Ravi Kumar, et al. · Computer Networks · 2,810

Web data mining: exploring hyperlinks, contents, and usage data

2012 · Choice Reviews Online · 1,179

Trawling the Web for emerging cyber-communities

1999 · Ravi Kumar, Prabhakar Raghavan, et al. · Computer Networks · 1,002

Reprint of: The anatomy of a large-scale hypertextual web search engine

2012 · Sergey Brin, Lawrence Page · Computer Networks · 878

Digital Methods

2013 · Richard Rogers · The MIT Press eBooks · 869

Efficient crawling through URL ordering

1998 · Junghoo Cho, Hector Garcia-Molina, et al. · Computer Networks and ISDN Systems · 844

PDF

Modern information retrieval: the concepts and technology behind search

2011 · Choice Reviews Online · 786

Statistics and Social Network of YouTube Videos

2008 · Xu Cheng, Cameron Dale, et al. · 735

Mining the Web: Discovering Knowledge from Hypertext Data

2002 · Soumen Chakrabarti · Thammasat University Digital Collections · 693

Crawling the Hidden Web

2001 · Sriram Raghavan, Héctor García-Molina · 674

Detecting near-duplicates for web crawling

2007 · Gurmeet Singh Manku, Arvind Jain, et al. · International World Wide Web Conference (WWW) · 616

Searching the Web

2001 · Arvind Arasu, Junghoo Cho, et al. · ACM Transactions on Internet Technology · 505

The indexable web is more than 11.5 billion pages

2005 · A. Gulli, A. Signorini · 412

Policy Response, Social Media and Science Journalism for the Sustainability of the Public Health System Amid the COVID-19 Outbreak: The Vietnam Lessons

2020 · Viet-Phuong La, Thanh-Hang Pham, et al. · Sustainability · 411

PDF

Large-scale incremental processing using distributed transactions and notifications

2010 · Daniel N. Peng, Frank Dabek · 408

Digital Methods

2013 · Richard Rogers · 387

Web Crawling

2010 · Christopher Olston, Marc Najork · Foundations and Trends® in Information Retrieval · 370

A large-scale study of the evolution of web pages

2003 · Dennis Fetterly, Mark Manasse, et al. · International World Wide Web Conference (WWW) · 360

Prying Data out of a Social Network

2009 · Joseph Bonneau, Jonathan Anderson, et al. · 339

Google's Deep Web crawl

2008 · Jayant Madhavan, David Ko, et al. · Proceedings of the VLDB Endowment · 339

Design and implementation of a high-performance distributed Web crawler

2003 · V. Shkapenyuk, T. Suel · Proceedings - International Conference on Data Engineering · 330

Data-Driven Cybersecurity Incident Prediction: A Survey

2018 · Nan Sun, Jun Zhang, et al. · IEEE Communications Surveys & Tutorials · 324

Introducing and evaluating ukWaC, a very large Web-derived corpus of English

2008 · Adriano Ferraresi, Eros Zanchetta, et al. · 319

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

2021 · Jesse Dodge, Maarten Sap, et al. · Conference on Empirical Methods in Natural Language Processing (EMNLP) · 300

PDF

Works count: 7,359

Citation count: 75,820