Work

Quantile and threshold calibration separate even for Gaussian forecasts

Year: 2026
Type: preprint
Abstract: Unrefereed research candidate. In one sentence: two standard ways of checking whether probabilistic forecasts are calibrated, quantile checks and threshold checks, can disagree even for smooth Gaussian forecasts, so passing one does not certify the other; this settles a conjecture of Gneiting and Resin (2023) and locates the exact boundary for outcomes with finitely many ordered values. The question. A probabilistic forecast is calibrated if its stated probabilities can be taken at face value. A threshold check fixes an outcome value x and asks whether, among forecasts that give the same probability F(x) of staying below x, that probability matches the observed frequency. A quantile (or conditional exceedance probability) check fixes a level α and asks whether, among forecasts with the same α-quantile, the outcome falls below it a fraction α of the time. For a smooth forecast, “F(x) = α” and “the α-quantile is x” pick out the same forecasts. Gneiting and Resin (2023, Theorem 2.16) proved the two notions equivalent when the forecast takes countably many values and conjectured that countability could be dropped; Resin, Yang and Gneiting (2026) still draw these implications as conjectured arrows. Results. (1) Countability cannot be dropped. Gaussian forecasts N(M, S2) with M uniform on [−1, 1] and S uniform on [1, 2], paired with explicit smooth outcome laws whose densities stay within a factor of two of the forecast densities, separate the notions in both directions: one law satisfies conditional exceedance and quantile calibration with an exactly uniform PIT, yet fails threshold and marginal calibration at every threshold; the other is threshold calibrated but fails quantile calibration, with a PIT distribution function strictly below the uniform one at every interior level. The errors reach about one percentage point: for example the outcome falls below 0 with probability 0.50758 while the average forecast probability is 1/2, which a binomial test at the 5% level with 80% power detects with about 34,000 cases. (2) For outcomes with k ordered values and positive forecast probabilities, conditional exceedance calibration implies threshold calibration when the forecast has countable support (any k) or k ≤ 3 (any forecast law), and fails for k ≥ 4 with an uncountable forecast family. (3) For nominal outcomes, full probabilistic calibration implies marginal calibration, with the sharp total-variation bound 2d − d2; class-wise calibration does not imply average probabilistic calibration (an example listed by Resin, Yang and Gneiting already indicates this; it is verified here, and an example with strictly positive probabilities is given). The nominal implications fCC⇒fTC and pCC⇒pTC hold for k ≤ 3 or countable support and remain open for four or more classes. (4) In location–scale families every compact perturbation preserving either property is described, with a general change-of-conditioning identity, transfer bounds and a local construction in any smooth forecast family. Why. Both checks condition on the same set of forecasts, a line in parameter space of probability zero, but weight it differently: the quantile check uniformly, the threshold check in proportion to the forecast's scale. An error that averages to 0 under one weighting averages to 1/9 (times the perturbation size) under the other: the Borel–Kolmogorov phenomenon of conditioning on zero-probability events. Limits. Population constructions, not a study of any deployed forecast system. The perturbation size 1/10 rests on a computer-assisted interval-arithmetic bound; the analytic proof gives 1/400. Two nominal implications remain open for four or more classes. Finite-sample test theory is not treated. Contents. The paper (LaTeX source and PDF) with its figure and figure script; three Python verification scripts (symbolic and numerical checks of the Gaussian examples and effect sizes; exact rational checks of the discrete and nominal results; an interval-arithmetic certificate of the amplitude bound) with recorded outputs; the bibliography from registry records; a machine-readable index; a research-gate record; a replay receipt and SHA-256 sums. Assurance. Written proofs, one computer-assisted interval certificate, and replayed checks. Internal AI-assisted review only (five reviewer roles, an adversarial review and audit by a second AI model, clean replays, one revision round, a re-review and a proof audit of new proofs): not peer review, independent reproduction or formal verification. Creator: Anonymous. Text and data CC0-1.0; code MIT.
Author: Anonymous
Language: English
Open Access status: green
OpenAlex ID: W7233582819
Cites: 15
Cited by:
Related to: