Noise, Selection and Speed: What Local Runs and a Public Leaderboard Can Tell You About a Coding Agent
Year: 2026
Type: preprint
Abstract: Agent competitions rank teams by the best of many stochastic runs on a few dozen hidden tasks. I measure how much such a ranking, and the local experiments behind it, can resolve, using a Gemma 4 31B coding agent in the Gemma 4 Developer Agent Competition. Three identical runs on 129 tasks solve 54, 51 and 53: 20 tasks (16%) change outcome between runs, which implies a run-to-run spread of 0.030 on the 58-task public score, close to the 0.025 seen in byte-identical resubmissions and well below the 0.047 a binomial model assumes. Under this noise, the best of k submissions can explain 0.06 of the 0.08 rise in median score from one-submission teams to teams with 32 or more. Local experiments have a second problem: runs that scale time limits by a calibration factor give the agent 2.5 times the time it has on the scorer. I confirm this on the scorer's own hardware, where the same agent's patches still pass at the local rate. The only significant gain in my study, 64 against 54 tasks with twice the call budget (p = 0.04), disappears at the measured speed. I give concrete rules for comparing agents, choosing final submissions and reporting leaderboard results.
Author: Malik, Raul
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225819921
Cites:
Cited by:
Related to: