Calibrated Against Which Label? What Temperature Scaling Certifies When Annotators Disagree, Why a Majority Vote and an Annotator Distribution Each Make the Other's Calibrated Model Look Wrong, and the Target a Calibration Claim Has to Declare
Year: 2026
Type: preprint
Abstract: (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Post-hoc calibration is usually fitted and scored against one label per item, and where annotators disagree that label is a majority vote. A 2026 study argues that this is a structural error: temperature scaling fitted to votes settles on temperatures that are too low, looks calibrated against the votes, and stays overconfident against the distribution of annotator labels, with errors that grow with annotation entropy; the best calibrator it fits to the annotator distribution, Dirichlet-Soft, cuts true-label calibration error by 55 to 87 percent. In this analysis the author reviews and synthesises published results to show that the finding is correct and that its framing is one-sided. Calibration is a relation between a forecast and a named random variable, and under disagreement there are at least four candidates: the vote, the label of a randomly drawn annotator, the label of a member of a target population, and an outcome fixed by later evidence. Each result in this literature is calibrated against a different one. A model that reproduces the human distribution exactly is badly miscalibrated against the majority, by the same logic that makes vote-fitted temperature scaling miscalibrated against annotators. The annotator distribution is not a clean target either: re-collecting 100 labels per item changed the majority label on a quarter to a third of low-agreement NLI items; adjudication traced most disagreements in one retinal grading study to missed lesions and artifacts; and the two medical benchmarks in the 2026 study use synthetic annotators whose confusion depends only on the consensus class. Re-reading that study's tables, the author finds that when temperature scaling is refitted to annotator labels, ECE falls by 48 to 87 percent while the soft-target Brier score falls by 2 to 11 percent. The paper consolidates the measured values in Table 1 and Figure 1, states a target-declaration procedure as Algorithm 1, and specifies six tests that would say which target a deployment should be calibrated to and how much of the measured gap survives it.
Author: Mahendrakar, Pranay
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234055479
