The Oracle Is Also on Trial: Who Verifies Ground Truth in Agentic Enterprise Benchmarks?
Year: 2026
Type: preprint
Abstract: Agentic enterprise benchmarks increasingly use executable tasks, hidden reference states, and automated judges to determine whether an AI system has completed consequential work. Yet an internally consistent benchmark can produce a false, unsupported, or institutionally inadmissible completion label when its task predicate, observation route, case identity, source lineage, authority, or target-world mapping is defective. This paper proposes the Oracle Independence Trial (OIT), a bounded, claim-relative protocol for auditing the warrant carried by a benchmark oracle rather than treating its reference label as unchallengeable truth. Building on outcome verification, evidence continuity, and adversarial cross-Field completion research, OIT distinguishes reference-world agreement, construct adequacy, source independence, real-world effect observation, and legitimate release authority. We specify typed support records, conditional propositions, counterexamples, and a prospective information-matched evaluation against strong conventional audit engineering. A deterministic reference demonstration generates 240 author-constructed cases across twelve failure families. A second, independently coded local audit reproduces its outputs, compares SQLite and Python adjudicators, checks 384 reduced states and 4,000 seeded perturbations, and exposes two limits: unbound cross-case witnesses can invert completion verdicts, while apparently well-addressed but false witness content can deceive both conventional and typed policies. An original assertion-based error-handling defect is disclosed. No live-agent experiment, independently verified oracle, measured Field-specific advantage, or production validity is claimed. The resulting contribution is a falsifiable framework for testing what justifies the authority of ground truth in enterprise-agent evaluations—and when that authority must remain explicitly unresolved.
Author: Figurelli, Rogério
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225621171
Cites:
Cited by:
Related to: