Get started
Work

Dependent Samples Break pass@k: An Impossibility Theorem for Unbiased Estimation, and the Estimand That Survives

Year: 2026
Type: preprint
Abstract: Search trees, planners and self-correcting agents make the attempts of one run share a random state, yet their results are often scored with a pass@k estimator built for independent draws, which public harnesses apply to whatever attempts they receive. Prior work separates the procedure's success probability from the independent-attempt one; we ask which of them such a number measures and how to report it validly, taking the complete run as the sampling unit. Whether a run succeeds within k attempts is an unbiased estimate of the procedure's success probability succ@k under any within-run dependence, and succ@k is the only quantity estimable from the first k attempts, with estimates in [0,1], that equals pass@k when attempts are independent. Pooling attempts across runs estimates a third quantity, which we give in closed form and which, for attempts conditionally i.i.d. within a run, lies between succ@k and the independent-attempt target pass@k^iid. Under a calibration condition and over a rich class of dependence structures, pass@k^iid has a uniformly unbiased estimator exactly when at least k complete runs are observed, however much is recorded inside each run, while per-question run-level confidence sequences are valid at any number of runs, for succ@k and, under the same condition, for pass@k^iid. In 8 designs spanning four dependence mechanisms, the pooled report exceeds the run-level estimate of succ@4 in every design, by +0.0205 to +0.1766, and on one design a run-level interval contains a fresh reference 0.952 of the time, against 0.104 for an attempt-level one. The fix is to name the target and count fresh runs.
DOI: 10.5281/zenodo.22675102
Language: English
Open Access status: green
OpenAlex ID: W7212047190
Cites:
Cited by:
Related to: