Work

Enhancing Scientific Knowledge Graph Generation Pipelines with LLMs and Human-in-the-Loop

Year: 2024
Type: conference-paper
Abstract: Scientific Knowledge Graphs have recently become a powerful tool for exploring the research landscape and assisting scientific inquiry. It is crucial to generate and validate these resources to ensure they offer a comprehensive and accurate representation of specific research fields. However, manual approaches are not scalable, while automated methods often result in lower-quality resources. In this paper, we investigate novel validation techniques to improve the accuracy of automated KG generation methodologies, leveraging both a human-in-the-loop (HiL) and a large language model (LLM)-in-the-loop. Using the automated generation pipeline of the Computer Science Knowledge Graph as a case study, we demonstrate that precision can be increased by 12% (from 75% to 87%) using only LLMs. Moreover, a hybrid approach incorporating both LLMs and HiL significantly enhances both precision and recall, resulting in a 4% increase in the F1 score (from 77% to 81%).
Authors: Tsaneva S., Dessi D., Osborne F., Sabou M.
Language: English
Open Access status: green
OpenAlex ID: W7019009455
FWCI: 0.462
Cites: 15
Cited by: 6
Related to: 10