Cogito, Ergo Feci: False Success, the Limits of Weight Surgery, and What an External Verifier Catches in a Computer-Use Agent
Year: 2026
Type: preprint
Abstract: Computer-use agents often fail by announcing that a task is finished when it is not. We diagnose this failure in Holo-3.1-4B on OSWorld and test whether editing activations or weights, without training, removes it. The agent scores 56.7% on 119 diagnostic tasks; a false "done" is 57% of its failures. With an input-exact replay and a gate fixed in advance, we find three traps that would each have produced a positive result: a false-success direction that adds at most 0.04 AUROC to position, run-to-run noise under which only 50% of false successes recur, and a loop direction in a smaller model that beats position yet reduces loops only at a large cost to healthy actions. No steering arm or weight edit passes the gate, and no edited or prompted arm run online beats the run-to-run noise of the original. The tool call leans on a thought that already declares success: neutral text in its place removes 80–82% of the answer margin. Of the edits that move the call, a model merge shows no detectable selectivity and a closed-form edit lowers false answers more, with an interval that includes zero. An external verifier that checks each requirement on the final screen detects 68% of false successes and rejects 10% of true ones offline (the agent judging itself: 53% and 48%); online, on 120 held-out tasks under a rule fixed in advance, it lowers the false "done" rate from 22.5% to 10.8% (−11.7 points, 95% CI −19.2 to −4.2) with no measurable change in score. Code, data and one-command verification of every number: https://github.com/martinbon39/ComputerUseHackaton
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225592947