Matched Twins Show What an Evidence Contract Cannot Carry to a Verifier of ML-Experiment Claims
Year: 2026
Type: preprint
Abstract: An ML-engineering agent reports that adding one engineered feature improved its validation score. A verifier bounded by the trace evidence (the claim, the reporting discipline, the outcome of plain re-execution, and the agent's one-line description of the change) must decide whether the gain is real. We construct matched twins: pairs of executed runs on the same task whose sentence, model types and budget block are identical, whose five-seed mean gains agree within 0.009, and which differ only in whether the added feature is a label leak. Scope-controlled re-execution separates the twins; the contract's fields carry almost no signal (the best single visible field reaches AUC 0.67, and it is the seed variance, which we did not manage to match: larger on the honest twin in 12 of 14 pairs). Two frontier reader models, 189 blind readers each on identical prompts, the primary reading fixed before dispatch: neither separates the pair. One flags leak and twin at 11/14 and 11/14 (exact p=1.0; bootstrap 95% interval on the difference -0.36 to +0.36), the other at 5/14 and 3/14 (p=0.69; -0.21 to +0.50); with 14 pairs the design can exclude only a gap above 0.43. A second, secondary result is an intervention on one field for one reader: the agent's sentence, which carries no truth value because the twins share it, moves that reader's verdicts on corrupt and honest traces alike. "Add one engineered feature" raises rejection of the leak (1/14→11/14) and of its matched twin (2/14→11/14); the tree-count sentence, worded truthfully for each class, lowers rejection of a starved baseline (18/18→4/18) and of an honest budget change (9/22→3/22). For the second reader the sentence does nothing; removing the budget block instead drops its rejection of the honest budget change (22/22→6/22) and of the starved baseline (17/18→9/18) and removes every unfair-baseline call, an ablation run after the replication and registered before dispatch. The replication did not hold as registered: two of its four readings replicated (twin non-separability and the legible classes) and two did not (the sentence effect and the honest false-positive bound). A test-retest of the primary reader two days later bounds what an unpinned reader can support: it agrees with itself on 25 of 46 prompts (kappa 0.15, below the registered bar of 0.6); the tree-sentence effect reproduces exactly (18/18→4/18 on both days) while its flag rate on the matched pairs falls from 11/14 to 3/14 with the sentence and to 0/14 without, so the feature-sentence effect keeps its direction and loses its size, and the same-day contrast reads indeterminate. A registered prediction that surfacing the labeler's two scope-controlled re-execution deltas inside the contract would make the pair legible failed: with a leak-free delta of 0.0 printed on every leak, the primary reader flagged the leak 0/14 against 7/14 in a same-day control (p=0.016), reading the field as confirmation of the gain. Re-encoded as a comparison, the plain gain beside the scoped gain and their gap, the same re-executions made the pair fully legible to the same reader (leak 14/14, matched twin 0/14, p=1.2 × 10^-4) and the starved baseline too (18/18), while turning the honest budget change into a false-positive class (3/22 to 17/22, fourteen called unfair on gaps of at most 0.01 or none at all). Replicated the same day with a fresh control and under the second reader, the separation held for both (14/14 to 0/14; 14/14 to 2/14) and the magnitude was read in opposite directions: the primary reader called the honest budget change unfair again (18/22), the second read the gap's size and stopped calling it unfair (20/22 to 6/22). The pair rates took a new value on each of four days while the budget classes reproduced task for task on all four. Under every encoding that omits the scoped re-execution the matched pair is non-separable on every day; under the one that carries it as a comparison it is separable on the day we ran it. We reached this measurement by catching ourselves three times, each time with a pre-registered blind pilot on the same corpus: a 0.909 false-positive rate that was a contract encoding defect, a leakage recall that swung from 0/21 to 15/21 on a corpus artifact, and a 0.90-against-0.59 twin separation that was claim magnitude. The corpus (242 executed traces, 22 tasks, 11 scripted proposers) is auto-labeled by counterfactual re-execution. A deterministic contract-bounded stand-in, not an LLM, reaches F1 0.82 at precision 1.0 on it while never firing on leakage, a class its rule set does not encode; in candidate selection the stand-in filter with the standard claimed-gain ranking falls to -0.02 of the oracle gap while re-executed credit closes 0.88. No funded API call was made; the readers are unpinned session models, and one reader model violated the no-tools instruction in two dispatches in a way we audit and disclose.
Author: Ko, Youngmin
Institution: Pennsylvania State University
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225147017