Assessing Answer Accuracy, Hallucination, and Document Relevance in a RAG-Based Chatbot at Osnabrück University
Year: 2025
Type: book
Abstract: This study evaluates a retrieval-augmented generation chatbot tailored for Osnabrück University, focusing on answer accuracy, hallucination, and the relevance of retrieved documents. Through human and automated evaluations conducted bilingually (German and English), it examines the chatbot's ability to deliver coherent, accurate, and contextually grounded responses to real user inquiries. Results highlight the chatbot's strengths in linguistic fluency and low hallucination rates but indicate variability in accuracy and context relevance. The study underscores the importance of hybrid evaluation methods combining automated metrics and targeted human assessments, offering insights into future refinement of domain-specific chatbots.
Source: osnaDocs (Osnabrück University)
DOI: 10.48693/655
Author: Wurch, Marvin Ives René
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6910919503
FWCI:
Cites:
Cited by:
Related to: