Get started
Work

Chatting with Papers - the mixed use of LLM's and semantic artefacts to support the understanding of science dynamics - and beyond

Year: 2025
Type: other
Abstract: Who would not love to have the possibility to navigate through a collection of documents (articles, datasets, websites) by asking all kinds of questions to it. The popularity of Google is long tangible proof of the need for search interfaces in these exploding information spaces. But, Google searches all over the web. Do we not often wish to have a ‘personal google’ for our own constructed knowledge bases or collections? Would this not be very useful for a better understanding of science dynamics, as this workshop asks? Information Retrieval is full of approaches to organise personal information spaces.[1] If it comes to exploring the past and present of science dynamics a toolbox for bibliometric analysis of formal scholarly communication is available.[2,3] Platforms have been also tested to collectively build, annotate, and analyse collections of scientific papers, targeting their metadata and the full text. [4] The new generation of LLM’s sparks new ideas to further improve information retrieval and large corpus analysis [7] also for the purpose of science studies. Our paper starts with the presentation of a new workflow (Ghostwriter [8]) that can be applied for information spaces locally defined. It continues with the demonstration of the use of such a new information retrieval tool to unravel hidden processes in the dynamics of the sciences [6]. More concretely, we report on a new development in the line of ‘chat or expert approaches’ which combines text analytic techniques (as used in LLM’s) with consulting structured data (as codified in controlled vocabularies). The workflow enables ‘chatting’ on both the metadata and data level for any collection of web resources (data collections, scientific paper collections and general web resources). It makes use of a recent metadata format standard (called Croissant for Machine Learning) for datasets used for machine-learning, and techniques to contextualise and semantically enrich information. [5]As a use case we present a collection of social science journal articles from the journal mda - methods, data and analysis. The journal was founded in 2007 and is published by GESIS - Leibniz Institute for the Social Sciences. It started with a mixture of German and English language articles ‘on on all questions important to quantitative methods in the social sciences, with a special emphasis on survey methodology’ to now be fully published in English as a Diamond Open Access journal. Using Ghostwriter on this collection allows to ask questions on both a close-reading-level and a collection overview level. Examples for the default use of a chat function are questions such as “What is a survey?” or “What is the response rate in surveys?”. Examples for the second type of questions are questions such as “Which are the most used methods/datasets in the article collection?” or “How do authors operationalise their research questions?” or “What is a typical survey sample size in accepted MDA studies?. The latter being typical for a science-of-science analysis. We demonstrate how the Ghostwriter approach can support human navigation through large information spaces for gaining orientation, overview, contextualisation and fact finding. We discuss how the combination of traditional bibliometric analysis with in-depth text-based conceptual analyses can lead to a better understanding of how scientific research questions emerge, how they are contextualised and how the social-communicative dynamics around them (e.g., scientific schools, invisible colleges) develops. In particular, we would like to discuss in the context of this workshop, how workflows such as Ghostwriter need to be shaped to support and empower the knowledge mastery of the human expert who is searching for a better understanding of science dynamics. References [1] Goharian, N., Nicola Tonellotto, Yulan He, Aldo Lipani, Graham McDonald, Craig Macdonald, Iadh Ounis (eds). Advances in Information Retrieval. 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24–28, 2024, Proceedings, Part I. Lecture Notes in Computer Science, Springer Cham; https://doi.org/10.1007/978-3-031-56027-9 [2] Van Eck, N.J., & Waltman, L. (2014). Visualizing bibliometric networks. In Y. Ding, R. Rousseau, & D. Wolfram (Eds.), Measuring scholarly impact: Methods and practice (pp. 285-320). Springer. (book chapter, preprint) [3] Mayr, P., & Scharnhorst, A. (Eds.) (2015). Combining bibliometrics and information retrieval: preface: Editorial. Scientometrics, 102(3), 2191-2192. DOI: 10.1007/s11192-015-1529-2 (Special issue) [4] Chavalarias, D., Lobbé, Q., Delanoë, A., 2021. Draw me Science: Multi-level and multi-scale reconstruction of knowledge dynamics with phylomemies. Scientometrics. https://doi.org/10.1007/s11192-021-04186-5 [5] Benjelloun, O., Simperl, E., Marcenac, P., Ruyssen, P., Conforti, C., Kuchnik, M., van der Velde, J., Oala, L., Vogler, S., Akthar, M., Jain, N., & Tykhonov, V. (2024, Mar 1). Croissant Format Specification. https://mlcommons.github.io/croissant/docs/croissant-spec.html [6] Scharnhorst, Andrea, Katy Börner, and Peter Besselaar (Eds.). (2012). Models of Science Dynamics. Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-23068-4. [7] Tong, X., Smirnova, N., Upadhyaya, S., Yu, R., Culbert, J. H., Sun, C., Otto, W., & Mayr, P. (2024). Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study. https://arxiv.org/abs/2408.13501 [8] Tykhonov, V., Rijsselberg, F. van ., & Endarto, E. (2024). The Next Generation of Data Management with Artificial Intelligence. Presentation, ODISSEI conference 2024, Utrecht, Netherlands. Zenodo. https://doi.org/10.5281/zenodo.14507120 Acknowledgement Parts of this paper have been funded by projects such as ODISSEI, SSHOC.nl, FAIRImpact and NFDI4DataScience. We thank Yves Rozenholc (University Paris Descartes) and the now.museum for collaboration and support.
DOI: 10.5281/zenodo.17242901
Language: English
Open Access status: green
OpenAlex ID: W7084431348
Cites:
Cited by:
Related to: