Year: 2026
Type: conference-paper
Abstract: While confidence estimation is a promising direction for mitigating hallucinations in Large Language Models (LLMs), current research dominantly focuses on single-turn settings.The dynamics of model confidence in multi-turn conversations, where context accumulates and ambiguity is progressively resolved, remain largely unexplored.Reliable confidence estimation in multi-turn settings is critical for many downstream applications, such as autonomous agents and human-in-the-loop systems.This work presents the first systematic study of confidence estimation in multi-turn interactions, establishing a formal evaluation framework grounded in two key desiderata: per-turn calibration and monotonicity of confidence as more information becomes available.To facilitate this, we introduce novel metrics, including a length-normalized Expected Calibration Error (InfoECE), and a new "Hinter-Guesser" paradigm for generating controlled evaluation datasets.Our experiments reveal that widely-used confidence techniques struggle with calibration and monotonicity in multiturn dialogues.We propose P(SUFFICIENT), a logit-based probe that achieves comparatively better performance, although the task remains far from solved.Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.Dataset Exemplar Prompt at Turn 4 20Q User: Given the following information, provide the title of the Wikipedia page that best answers the last question fragment.If unsure, provide your best guess.The answer should be concise.You have some clues about the answer: Assistant: Is it human-made?User: Yes Assistant: Is it typically found indoors?User: Yes Assistant: Is it commonly encountered in living rooms?
Institutions: University of Cambridge, Fudan University
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166832105