How Much Does a Parity Experiment Prove?
Year: 2026
Type: preprint
Abstract: Harbor, the evaluation framework behind Terminal-Bench, checks benchmark ports by comparing run scores from the original harness and the port. A port passes when the score ranges overlap. With three runs per side, exchangeable scores under parity, and no ties, the rule flags a perfect port 10 percent of the time. Adding runs makes it less likely to detect a fixed gap. In 58 published comparisons with three runs per side, 57 pass range overlap and only 22 establish equivalence within 3 points. Under normal noise at the median observed variability and a true gap of zero, an equivalence test needs about seven runs per side for 80 percent power at that margin. In 14 of 27 adapters, the two sides disagree on individual tasks more than shared pass probabilities predict at an unadjusted 5 percent threshold. Mean-gap tests find no statistically detectable difference in those adapters. With ten runs per side, range overlap detects a simulated defect that breaks 10 percent of tasks in 36 percent of cases. The pooled and stratified tests each detect it in 86 percent. I propose a parity protocol and provide a verdict tool.
Author: Barve, Poorva
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225640380
Cites:
Cited by:
Related to: