Work

Evaluating Open-Source LLMs in RAG Systems: A Benchmark on Diploma Theses Abstracts Using Ragas

Year: 2025

Type: article

Abstract: Retrieval-Augmented Generation (RAG) systems have gained significant attention for their ability to enhance language models by incorporating external knowledge sources. However, evaluating the effectiveness of these systems remains a challenge, as both the retrieval and generation components must be assessed independently and in combination. This study aims to provide a comprehensive evaluation of open-source large language models (LLMs) within a RAG system, using a dataset derived from diploma theses abstracts. To systematically assess performance, we generated a benchmark dataset consisting of 122 questions with reference answers, categorized into reasoning, fact-based, and summary-type questions. Using Elasticsearch as the retriever and several open-source LLMs as generators, we evaluated the system using the Ragas (Retrieval Augmented Generation Assessment) framework, focusing on retrieval effectiveness and answer quality. Our findings highlight the strengths and weaknesses of different models and retrieval strategies, offering insights into optimizing RAG implementations for academic and structured knowledge tasks. Our dataset and the corresponding evaluation code are publicly accessible at https://github.com/margitantal68/rag_paper .

Source: Acta Universitatis Sapientiae Informatica

DOI: 10.1007/s44427-025-00006-3

Authors: Margit Antal, Krisztian Buza

Institution: Budapest University of Economics and Business

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $3,190

Funder: Universitatea Sapientia