Viveka: Separating Real Gains from Chance in Self-Improving Agents
Year: 2026
Type: preprint
Abstract: Agents that improve their own harness usually keep a change when a score goes up. That is weak evidence. Agent evaluations are noisy, and an agent that tries many changes will keep some that only looked better by chance. We show two simple facts. Under the usual "keep it if it scores higher" rule, a change that does nothing is kept half the time, no matter how many seeds are used. And once such a change is kept, its lucky score becomes the bar that real improvements must clear, so real progress slows down. We propose Viveka, a checking layer that sits outside the agent. It re-tests each change on fresh seeds the agent has never seen, stops early when the answer is clear, replaces lucky scores with fresh ones, and later removes changes that turn out not to matter. In a simple simulation of 100 proposed changes, the usual rule reports a gain of 1.92 sigma when the real gain is 0.52 sigma. Viveka reaches a real gain of 1.43 sigma, while spending a similar budget on extra seeds reaches 0.82 sigma. These numbers come from a simulation, not from real agents. We end with a fixed plan for testing the same claims on an open-source self-refining coding agent.
Author: Chowdhury, Sayan
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225819651
Cites:
Cited by:
Related to: