Year: 2025
Type: dissertation
Abstract: Assessing candidate knowledge through free-text answers poses significant challenges for automation, often hindered by the scalability and consistency limitations of traditional methods like manual grading or simple keyword matching. In this thesis, we investigate an application workflow leveraging Large Language Models (LLMs), enhanced by the Retrieval-Augmented Generation (RAG) framework, for automated knowledge validation against domain-specific documents. We present a system architecture where Retrieval Augmented Generation (RAG) is utilized to dynamically retrieve relevant factual passages from a designated knowledge corpus to provide context to an LLM, which then evaluates the correctness and contextual relevance of a candidate's answer. We implement and evaluate (via LangChain and Ollama open-source libraries) an application to assess the practicality of this approach given hardware constraints imposed by commercial computers. Findings indicate that the RAG-enhanced system significantly improves factual grounding and contextual understanding compared to standalone LLMs, potentially allowing the use of smaller, more resource-efficient models. We also spot critical challenges, including increased system complexity, processing latency due to the retrieval step, slow document indexing, and a crucial dependency on the quality and comprehensiveness of the knowledge base. While RAG-enhanced LLMs present a promising advancement for automated knowledge evaluation, successful deployment requires balancing the benefits with these practical considerations.
Source: Doria (University of Helsinki)
Author: Bitarafan, Ali
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7115289017