isitright: Auditing Passing Evals by Changing the World They Run In
Year: 2026
Type: preprint
Abstract: A passing eval checks an agent or query in one world: one database, one set of bookings, one clock. It leaves open how it behaves when that world changes. We introduce isitright, an open-source tool built on an extensible framework for strengthening users' own passing agent evals. It searches for valid world changes exposing disagreements with a reference checker, or oracle. Users supply the environment and reference; a shared core generates changes, checks their validity and records evidence that can become new eval cases. Domain adapters support fixed SQL queries and live tool-using agents, including agents whose behavior varies between runs. In an airline case study, an agent agrees with the reference in 115 of 116 judged original-world runs, but generated cancellation worlds show increased disagreement on eight of nine tasks. A second agent reaches at least 75% observed agreement on five of those worlds. A SQL case study finds 14 query disagreements satisfying database structural checks. These are findings for review, not established agent defects: reports distinguish world validity, the reference's assumptions, execution conditions and evidence from repeated runs.
Author: Pogrebnyakov, Nick
Language: English
Subfield: Software
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225129457