The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
Year: 2026
Type: preprint
Abstract: You have 25,000 measurements from 500 sources. How many independent measurements are they worth? Not 25,000, and not 500. Those two answers are the endpoints of a single law: no correlation within a source at one end, perfect correlation at the other, and both endpoints are already known. What had not been derived is the interior, which is where every practitioner actually sits. Measured on one released system, the answer is about 1,300. A predictor calibrated on 25,028 scores drawn from 500 reasoning traces carries the reliability of roughly 1,300 independent examples, and the spread of the coverage it actually delivers is 4.46 times wider than independence implies. The question is not specific to machine learning. Engineering teams set p99 latency objectives from request logs whose requests cluster by user, session or host. Radiologists set diagnostic thresholds from images that arrive several to a patient. Laboratories set reference intervals from national surveys whose participants arrive in geographic clusters, and paediatric growth standards set the cutoff defining stunting at an estimated centile of children recruited at six sites. Risk desks backtest value-at-risk against counts of exceedances. Each of these sets a decision threshold at a sample quantile of clustered data, and the standard correction for clustering, Kish's design effect of 1965, is built for means. A threshold set at a sample quantile is an order statistic, not a mean, and an order statistic cannot see how similar two scores are; it sees only which side of the threshold each falls on. Two same-source scores of 0.82 and 0.79 act as one point when the threshold is 0.9 and as two independent points when it is 0.8. So the correlation Kish's formula needs is not the score correlation anyone measures. It is the correlation of the exceedance indicators at the operating level, written rho_I(p). For b independent families of within-family exchangeable scores, coverage conditional on the calibration set is asymptotically normal with variance p(1-p)/n_eff, where n_eff = n / [ 1 + (m_tilde - 1) rho_I(p) ] and m_tilde is the size-biased mean family size. Three factors, each separately estimable: a dependence measure that moves with the coverage level, a size measure that is not the average family size, and the raw count. The density cancels in the proof, so the law depends on the score distribution only through the copula diagonal, and it recovers the i.i.d. Beta law at zero correlation and the family-count law at perfect correlation, closing an interpolation named as an open problem in 2023 and left underived by two 2026 preprints. The machinery is classical, and Section 7 tabulates who owns each ingredient. What is new is the composition: a level-dependent correlation and a threshold estimated from the very sample it governs, priced as a distribution over realised coverage rather than a bound on its mean. Three consequences. (I) The design effect written down in recent conformal work under dependence, Kish's formula fed the score correlation, is the wrong quantity, and it is not wrong conservatively. On a valid copula whose score correlation is exactly zero, so that the correction vanishes entirely, the true design effect is 1.44. The same inversion occurs on real released data. (II) rho_I moves with the level, so no system has one design effect: 1.97 at 90% coverage against 1.54 at 99%, at identical score correlation. A system has a curve, not a number. (III) Because the dispersion penalty is first order while the mean bias is only O(1/b), the damage is nearly invisible in the statistic these papers report. A method evaluated by averaging coverage over many runs reports a number that is very nearly correct and wholly uninformative. The practitioner who deploys once gets a draw. Consequence (III) is counted rather than argued. Mean coverage averages over calibration draws that no user ever meets; a deployment draws its calibration set once and lives with it. On the released set, mean coverage moves by 0.03 percentage points while the rate at which a deployment misses its promised level by half a point moves from about 4 per 1,000 to about 285. A second calibration set, generated by us at two decode widths, isolates the mechanism to shared prefixes. Beam-search siblings carry an exceedance correlation an order of magnitude above independently sampled siblings of the same prompts, and widening the beam raises the design effect through family size while the per-unit dependence falls. Someone who widens their beam, watches the design effect climb and reports rising dependence has reported a change in family size. The released artifact also carries a violation larger than the one characterised here, which is reported against our own interest. Family size is coupled to family score: harder questions produce longer traces, hence more prefixes and lower success rates, and that breaks an assumption the theorem itself needs. What it costs turns on a choice the system's authors never state. Under one reading of which unit a future test point is, it moves coverage by 4.99 percentage points, 69 times the mean effect the design effect produces on the same data; under the other reading it is exactly zero. A design-effect correction applied to such data would be a precise answer to the wrong question. 46 pages. Verification code (83 files) accompanies this record and is mirrored at github.com/ACNoonan/exceedance-design-effect. A standalone ten-page statement of the theorem, with proof and verification recipe, is attached as its own PDF and compilable .tex. Cite the concept DOI 10.5281/zenodo.21595640, which always resolves to the newest version. What's new in v7 Nineteen registered claims had moved since v6, so every number below was already wrong for a v6 reader before this version was cut. The corrections that change what a reader would do come first, then the restructuring. Sampling-depth budget, Sections 6 and 8. The limit on what more samples per source can buy was computed by holding the dependence parameter fixed while the sources grow. Section 6.2 of the same paper measures that parameter falling when they grow: quadrupling family size lowers it by 12 percent, with disjoint confidence intervals. The premise was refuted by the paper's own experiment and the consequence was understated by roughly eighteen-fold. Ten times as many samples per source buys about 220 effective points, not twelve, and there is no finite ceiling at all, only a slow divergence. The recommendation is unchanged and now rests on a measured response rather than an assumed invariance: buy breadth, not depth, by a factor of about four rather than sixty. Training-conditional guarantees, Section 8. The published correction for a training-conditional coverage guarantee is Hoeffding's inequality, which requires independent calibration points. Applied unchanged to the clustered released set it does not merely lose sharpness; it returns 0.737 where 0.90 was required, so the guarantee it was invoked to buy is void. Hoeffding does apply at the level of sources, where the effective count is n divided by the size-biased mean and the shift inflates by its square root: 408 rather than 25,028 on this artifact, a factor of 7.83. The corrected form is distribution-free in the dependence, so a practitioner with no estimate of the correlation can still use it. It is loose by a factor of three to six, and that is stated where it is used. Three citation corrections. Two 2026 papers entered as Anonymous have public author lists on arXiv: CROP is by Cheung, Veeraraghavan, Chen and Balakrishnan (arXiv:2605.30085), and Conformal Path Reasoning is by Lin, Zhou, Lin, Dong, Lu, Peng, Yin and Metaxas (arXiv:2605.08077). A third entry carried the paper's own title in its author field, so the PDF cited a work authored by "Conformalized Lee inference"; the paper is by Jung Hyub Lee (arXiv:2607.02898). A quotation from Conformal Path Reasoning also read "restore" where the source says "preserving", which is a different claim, and is corrected. Restructured, and twelve pages shorter, with no printed number moved and no claim weakened. The coverage law now directly follows the introduction, and related work follows the result it differentiates. Attribution that was repeated across sections is stated once at the point of first use, with Section 7.1's ownership table as the canonical record. The thirteen-system exchangeability audit has been removed: it examined how systems source the assumption in general rather than anything specific to the exceedance design effect, and the two measured systems remain as Section 6. A mean-target estimator comparison on public benchmarks was out of scope for the same reason and was removed, recorded in the claim ledger as removed rather than refuted. The limitations section is retitled and shortened to the two axes that belong in it: what is not proved, and what is not measured. Deployment rates, Section 6. The measured coverage range is now also reported as the rate at which a single deployment misses its promised level, with a negative control: the same code path on the same released questions, with the clustering removed by construction so that every source contributes one unit, returns a dispersion ratio of 1.002 against the reference law. That control is informative only because a positive control gates it. The clustered arm must first reproduce the published 4.46-fold inflation, since a harness that inflated nothing would also pass the negative control. Four goodness-of-fit figures, Section 6.1. The three candidate coverage laws were scored against a Beta parameterised in a form the paper does not use, which distorted the comparison rather than shifting all arms together. Re-scored against the paper's own form, the plug-in law's distance moves from 0.0965 to 0.0710, or 36 percent, while the naive and matched arms barely move. No claim reverses.
Source: arXiv (Cornell University)
Author: Noonan, Adam
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223764944