Truthfulness Under Prompts: Systematic Evaluation of Hallucinations in Llms With Truthfulqa
Year: 2025
Type: dissertation
Abstract: Large language models (LLMs) have achieved remarkable progress in Natural Language Processing (NLP), significantly enhancing the performance of core NLP tasks and enabling breakthroughs in various application domains.However, recent LLMs remain prone to generating hallucinations-statements that are not grounded in real-world facts or input data-which poses challenges for the reliability of model outputs.To systematically assess the truthfulness of LLMs, we leverage the TruthfulQA benchmark, which is adversarially designed to test models' susceptibility to imitative falsehoods.In this study, we comprehensively analyze the ability of state-of-the-art LLMs to produce truthful responses under different prompting strategies.Additionally, we extend the evaluation to the MC1 and MC2 tasks officially specified in the TruthfulQA benchmark, which require the identification of single or multiple correct answers from a set of options.Our results demonstrate that a novel prompting approach-by encouraging or even misleading the model-can substantially improve the average truthfulness of LLM-generated responses.These findings offer valuable insights for the development of more reliable and trustworthy language models.
Author: Zheng, Xuyi
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7208513455