The Gap Between Self-Reported Success and External Acceptance: A Single-Case Study of an AI Agent with a Prediction Ledger
Year: 2026
Type: preprint
Abstract: When an AI agent reports that a task succeeded, how much of that report has been independently checked? We present a single-case study of a locally deployed coding agent equipped with a prediction ledger: before any modification, the agent must record a bet consisting of four questions (what will succeed, how it will fail, how failure would be detected, and how success would be recognized). A separate agent then evaluates whether the stated acceptance criterion is falsifiable. Across 147 recorded bets (116 settled), the agent's own success rate is 104/116 = 89.7%, while only 16 of the 116 settled bets were ever externally reviewed (16/116 = 13.8%); of the 25 bets that entered review at any time, 16 were rejected (16/25 = 64.0%). We further report that the measured coverage ratio shifts on a six-minute scale, because its numerator grows with the sample while its denominator does not, and we argue that this instability is a property of the metric rather than a defect of this system. We find no evidence that the ledger improves outcomes, because no control condition was run. We accordingly frame this as a negative and diagnostic result: the contribution is a measurement of how large the unchecked fraction of agent self-assessment can be, and a warning that a success rate computed over self-reported outcomes is only informative to the extent that the reviewed subset is representative of the unreviewed one. We did not sample the unreviewed cards at random and did not record the selection rationale for reviewed cards, so we can establish neither the direction nor the magnitude of that non-representativeness.
Author: Guo, ShuQing
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225433681