Work

Chaos Monkey for Coding Agents: Measuring How Coding-Agent Harnesses Recover When Their Tools Fail

Year: 2026
Type: preprint
Abstract: Coding agents run inside harnesses that execute their tool calls, and those tools fail: commands error, reads return nothing or stale data, and calls that succeeded report failure. We present Chaos Monkey for Coding Agents (CMCA), an open benchmark that applies one seeded fault model at two layers: an MCP proxy in front of mock services, and injectors loaded through each harness's own hooks that fault its built-in read, edit, search and shell tools, restricted to fault types a conformance suite shows each harness delivers. Every configuration uses the same main model, so rows compare harness configurations; the native-track fx row is fxh, a fork that adds tool hooks. Besides success under faults, CMCA reports whether agents state their outcome honestly, whether stale data reaches their output, and what recovery costs. On 37 service and 36 native tasks (3,011 trial runs), fx leads the service track (83 of 111 faulted trials) and fxh the native track (95 of 108), but no pairwise difference survives Holm correction. Among failed trials that received a fault, agents still reported success in 60% to 94% of cases. Code, tasks, trajectories and analysis are public. Results are release 2026-11-18-m6 (git tag m6-2026-11-18). Results site: https://www.chaos-monkey.tech
Language: English
Subfield: Software
Open Access status: green
OpenAlex ID: W7237246720
Cites:
Cited by:
Related to: 10