CRAWLDoc: A System for Contextual Ranking and Bibliographic Metadata Extraction from Web Resources
Year: 2026
Type: article
Abstract: Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This paper introduces CRAWLDoc, a novel system for contextual ranking of linked documents and metadata extraction from web resources. Using a publication’s DOI, CRAWLDoc retrieves the landing page and associated linked documents, including PDFs, ORCID profiles, and supplementary materials. It embeds these documents, along with anchor texts and URLs, into a unified representation. Our layout-independent embedding and ranking system ensures robustness across various web layouts and formats. Experimental results demonstrate CRAWLDoc’s superior performance in extracting bibliographic metadata compared to relying solely on landing pages. A leave-one-out experiment across six publishers shows CRAWLDoc’s resilience to layout changes and consistent extraction accuracy. Our source code and dataset can be accessed at https://github.com/FKarl/crawldoc-metadata-extraction
Source: Information Retrieval Research
DOI: 10.54195/irrj.23861
Authors: Fabian Karl, Ansgar Scherp
Institution: Universität Ulm
Topic: Web Data Mining and Analysis
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Deutsche Forschungsgemeinschaft
OpenAlex ID: W7127588879