Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
Year: 2026
Type: conference-paper
Abstract: Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted.However, common approaches such as verbalization produce extremely sparse outputs.For instance, Qwen3-32B verbalizes only eight unique confidence values on SST-2, with over half being exactly 95%-a pattern we observe consistently across four datasets and two LLMs.Besides limiting practical utility, we show that this sparsity critically affects evaluation: the choice of interpolation in area under the accuracy-rejection curve (AUARC) dramatically alters rankings, with consistency sampling dropping from best to worst under stepwise versus linear interpolation.We advocate for standardizing stepwise interpolation for a fairer comparison.Under such a fair evaluation, we find that weighting verbalized digits by token probabilities-a method we term verbalization logprobs-addresses sparsity and achieves the best AUARC (+2.3 points over vanilla verbalization) without incurring additional inference cost.
Institution: Humboldt-Universität zu Berlin
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: Deutsche Forschungsgemeinschaft
OpenAlex ID: W7166803155
FWCI:
Cites:
Cited by:
Related to: