HTMLDownloader: An open-source tool for dynamic web scraping and archiving using WebView2
Year: 2025
Type: software-paper
Abstract: The increasing complexity and dynamism of modern websites present major challenges for traditional web scraping tools such as Scrapy, BeautifulSoup and wget, which often fail to capture dynamic content or offer accessible user interfaces. To address these limitations, we introduce HTMLDownloader which provides a graphical interface (GUI) that makes it accessible to non-technical users and enhances scraping reliability by integrating browser-based rendering. Experimental evaluations on 75,516 links from 35 diverse domains demonstrate a 98.4% success rate, significantly outperforming Selenium (81.3%), Scrapy (56.6%), BeautifulSoup (34.5%) and wget (45.1%). These results confirm HTMLDownloader’s robustness and scalability, making it a powerful solution for dynamic content extraction and long-term web archiving. HTMLDownloader ships as end-user MSI/portable ZIP with documented workflows, enabling non-specialists to reproducibly archive JavaScript-heavy pages. A DOI-tagged release supports verification, reuse and citation (DOI: 10.5281/zenodo.16935169).
Source: SoftwareX
Authors: Ba-Vinh Truong, Loan T.T. Nguyen, Phu Pham, Bay Vo
Language: English
Topic: Web Data Mining and Analysis
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,200
OpenAlex ID: W7083053435