Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in NLG
Year: 2025
Type: conference-paper
Abstract: Hallucinations are one of the most pressing challenges for large language models (LLMs). While numerous methods have been proposed to detect and mitigate them automatically, human evaluation continues to serve as the gold standard. However, these human evaluations of hallucinations show substantial variation in definitions, terminology, and evaluation practices. In this paper, we survey 64 studies involving human evaluation of hallucination published between 2019 and 2024, to investigate how hallucinations are currently defined and assessed. Our analysis reveals a lack of consistency in definitions and exposes several concerning methodological shortcomings. Crucial details, such as evaluation guidelines, user interface design, inter-annotator agreement metrics, and annotator demographics, are frequently under-reported or omitted altogether.
Authors: Schmidtová, Patrícia, Calò, Eduardo, Balloccu, Simone, Gkatzia, Dimitra, Huidrom, Rudali, Lango, Mateusz, Same, Fahime, Zouhar, Vilém, Mahamood, Saad, Dušek, Ondřej
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7111967299
FWCI:
Cites:
Cited by:
Related to: