Using LLMs for Automatic Discipline Annotation in a Diachronic Corpus of English Scientific Papers
Year: 2026
Type: conference-paper
Abstract: This study investigates the potential of generative large language models (LLMs) to automatically identify the disciplines of scientific papers in the Royal Society Corpus (RSC) -an extensive collection of English scientific publications spanning more than three centuries.We evaluated eight open-source, state-of-the-art LLMs from four model families on a manually annotated subset and further validated the three best-performing models on a corpus of modern scientific texts.These models were subsequently used for large-scale annotation of the RSC.The models exhibited robust and consistent performance, with at least two LLMs agreeing on the same label for 98.3% of the documents.We then conducted an error analysis of papers assigned divergent labels and a diachronic case study of disciplinary trends within the corpus.The error analysis revealed that most discrepancies occurred in twentieth-century texts, reflecting the growing interdisciplinarity of research.The diachronic analysis showed a gradual decline in disciplinary diversity over time as well as fluctuations corresponding to major paradigm shifts such as the Chemical Revolution and key twentieth-century developments in Physics.The discipline labels generated by the three models will be made publicly available.
Institution: Saarland University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7160024471