The judge cannot see the margin: a paired ablation of a mechanical proof step in a model-run design pipeline
Year: 2026
Type: preprint
Abstract: Desk Studio runs a design job as a loop of model calls: a builder polishes a card, a browser renders it, a proof step measures the rendered pixels, and a Creative Director scores the screenshots against a rubric and signs off or sends the round back. Proof is the one step that never calls a model. I asked whether it changes anything when a strong vision model is already looking. I added a flag, --no-proof, that skips the checks and leaves everything else in place, and ran 15 cases in both arms and in two model families (claude-opus-5-5 and gpt-6-astra as directors), 60 jobs in 30 pairs, with four hypotheses written down before any job finished. Without proof the director signed off 17 files and all 17 failed a hidden audit. With proof, 0 of 17. Against my prediction, proof cost less, 0.735 against 0.985 USD a job at list price in the Anthropic family (exact p = 0.064) and 1.074 against 1.311 pooled (p = 0.021), because a failed proof round costs a builder call and not a director call. The director's final rubric averages did not differ, 8.49 with proof and 8.43 without (p = 0.307), so its scores carry no trace of the faults. Without proof it named a mechanical fault in 13% of its notes, against 1% with proof, and still signed the faulty files. In 19 of the 30 proof-on jobs a later polish round broke a check an earlier round had passed. The rule I take from this is that anything measurable belongs in a check on the rendered file, run after every round, and the judge keeps what only it can see.
Author: Reddy, Aditya
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224305752