The blueprint of a new fact-checking system: A methodology to enrich RAG systems with new generated datasets
Year: 2025
Type: article
Abstract: In an era where digital misinformation spreads rapidly, Artificial Intelligence (AI) has become a crucial tool for fact-checking. However, the effectiveness of AI in this domain is often limited by the availability of high-quality and scalable datasets to train and guide algorithms. In this paper, we introduce VERIFAID (VERIfication FAISS-based framework for fake news Detection), a novel framework that improves fact-checking through a Retrieval-Augmented Generation (RAG) system based on automatically generated and dynamically growing datasets. Our approach improves evidence retrieval by building a scalable knowledge base, reducing the reliance on manually annotated data. The system consists of three key modules: two dedicated to dataset creation and one inference module that integrates advanced language models, such as LLaMA, within the RAG paradigm. To validate our methodology, we provide technical specifications for both the system and the dataset, together with comprehensive evaluations in zero-shot fact-checking scenarios. The results demonstrate the efficiency and adaptability of our approach and its potential to improve AI-driven fact verification at scale. • A RAG-based approach can compete with and outperform other fact-checking methods. • An auto-updating database enables real-time use and reduces human intervention. • Adding metadata and embedding distance improves classification in LLM contexts. • LLM behaviour improves with role specification and the inclusion of more chunks. • LLMs effectively extract claims and verdicts when data is in natural language.
Institution: Universidad de Granada
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $2,880
Awards: G210327001, G5035208470, G5091461029
OpenAlex ID: W4414982232