Building a Reliable Retrieval-Augmented Generation (RAG) System with Hallucination Awareness
Year: 2026
Type: dissertation
Abstract: Retrieval-augmented generation has become an important approach for improving the factual grounding of large language model outputs by conditioning generation on externally retrieved documents. However, retrieval alone does not guarantee reliable answers. A system may retrieve irrelevant or incomplete evidence, generate unsupported claims, or fail to communicate uncertainty when the available knowledge is insufficient. This thesis investigates how RAG systems can be designed and evaluated with a focus on reliability and hallucination-aware behaviour in a fixed-size knowledge base. The work follows a system-building and evaluation approach. A modular RAG prototype was developed to support controlled experimentation across different pipeline configurations. The system was evaluated on a researchpaper corpus using a custom query dataset containing answerable, ambiguous, and unanswerable questions. Three main retrieval configurations were compared: baseline dense retrieval, query expansion with reranking, and dynamic retrieval based on reranker score gaps. The experiments also compared three embedding models and multiple retrieval depths. The evaluation used retrieval-side metrics, including recall, hit rate, and mean reciprocal rank, together with generation-side metrics such as answer correctness and faithfulness. The results show that retrieval-side design choices substantially affect system reliability. The choice of embedding model had a strong impact on retrieval quality, while query expansion improved evidence coverage and ranking, especially under smaller retrieval depths. Dynamic retrieval provided a strong trade-off between retrieval quality and context size by selecting a variable number of chunks while maintaining competitive performance. However, the experiments also showed that stronger retrieval metrics did not always lead to proportional improvements in answer correctness or faithfulness. Qualitative error analysis further showed that reliability failures included not only unsupported answers, but also unnecessary refusals and insufficiently calibrated responses to ambiguous questions. The thesis concludes that hallucination-aware RAG should be treated as a pipeline-level problem. Reliable behaviour requires not only retrieving relevant evidence, but also assessing whether the evidence is sufficient for the specific question and communicating uncertainty when appropriate. The developed prototype and evaluation provide a practical foundation for further work on adaptive, transparent, and trustworthy RAG systems.
Source: Aaltodoc (Aalto University)
Author: Pajaziti, Loreta
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203956628