Year: 2025
Type: conference-paper
Abstract: We study the source of uncertainty in DeepSeek R1-32B by analyzing its self-reported verbal confidence on question answering (QA) tasks.In the default answer-then-confidence setting, the model is regularly over-confident, whereas semantic entropy-obtained by sampling many responses-remains reliable.We hypothesize that this is because of semantic entropy's larger test-time compute, which lets us explore the model's predictive distribution.We show that granting DeepSeek the budget to explore its distribution by forcing a long chain-of-thought before the final answer greatly improves its verbal score effectiveness, even on simple factretrieval questions that normally require no reasoning.Our analysis concludes that reliable uncertainty estimation requires explicit exploration of the generative space, and self-reported confidence is trustworthy only after such exploration.
Authors: Jakub Podolak, Rajeev Verma
Institution: University of Amsterdam
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4416033781