Work

An Adaptive Asynchronous Web Scraping Framework for Concurrent Extraction of Links, Boilerplate-Free Text, and Documents using JavaScript-Aware Rendering with OCR Fallback

Year: 2026
Type: article
Abstract: The exponential growth of online information has made automated web data extraction essential for research, business intelligence, and digital archiving. Conventional scrapers, however, typically rely on sequential request handling and static HTML parsing, which limits throughput and fails on modern JavaScript-rendered pages and scanned documents. This study presents an adaptive, asynchronous web scraping framework that unifies five extraction capabilities—embedded-link harvesting, boilerplate-free page-text extraction, combined text-and-link extraction, text extraction from linked PDF documents, and bulk download of PDF and image assets—within a single pipeline. The framework introduces four improvements over the conventional approach: concurrency-limited asynchronous fetching, an adaptive static-versus-dynamic rendering heuristic that escalates only JavaScript-dependent pages to a headless browser, boilerplate-aware main-content extraction, and an optical character recognition (OCR) fallback for scanned PDFs. The framework was evaluated on scraping-permitted benchmark sites with statistical replication and an independent gold standard. Over five trials, asynchronous downloading reduced the time to retrieve 40 assets from 41.2s (SD 5.8) to 2.28s (SD 0.11), a mean speed-up of 18.1 times. Across three JavaScript-rendered pages, adaptive rendering increased recovered text by a mean factor of 23.6. Measured against the Wikipedia plain-text gold standard, boilerplate-aware extraction raised main-content precision from 0.548 to 0.687 while maintaining recall near 1.0, improving F1 from 0.703 to 0.811. The OCR fallback recovered text from scanned PDFs that yielded no characters under conventional parsing. The results demonstrate that the framework is a faster, more robust, and higher-quality solution for web-scale data extraction, while its limitations under timer-delayed content are openly characterized.
Language: English
Open Access status: bronze
OpenAlex ID: W7225201210
FWCI:
Cites: 7
Cited by:
Related to: 10