Run the Real Tests: Scoring every intermediate step of 345 Gemma 4 coding-agent runs
Year: 2026
Type: preprint
Abstract: A coding agent edits a repository many times before it submits a patch, but only the submitted patch is scored. We score all of the steps in between. We record the workspace after every file-changing action and run the competition’s official hidden tests on each version. The data cover Gemma 4 E4B running the official sample agent on 72 valid tasks, and 273 Gemma 4 31B runs from 17 agent configurations, rebuilt from public logs. In total that is 1,431 distinct intermediate states, all scored. Three results follow. First, agents seldom lose a fix they already had (2 of 345 runs), so checkpoint-and-undo machinery has little to recover. Second, the agent’s own sense that its fix works is unreliable: only 48% of 31B’s passing test runs coincided with a correct workspace, and 29% for E4B. Third, the kind of check matters. A pass from the repository’s own test suite was right 68% of the time, while a pass from a scratch reproduction script was right 38% of the time (Fisher p = 2×10−8). For people building small agents, the cheapest improvement is to make the agent confirm its fix with the project’s real tests, and not to trust an early green result.
Author: Shahi Thakuri, Sundar
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225571569