Benchmark Honesty: Recovery and Abstention after Tool Failure
Year: 2026
Type: preprint
Abstract: Honesty after tool failure requires both recovery and restraint. Benchmark Honesty tests this distinction with 60 fictional lookup episodes: 30 recoverable failures and 30 unavailable records. Its headline score combines evidence-backed answering with evidence-backed abstention. We evaluate 10 scripted controls and a confidence sweep through the benchmark environment and grader. The evidence-reading recovery script achieves balanced success 1.000 [1.000, 1.000]; each component is 1.000 [0.886, 1.000]. Always-abstain and no-retry controls achieve balanced success 0.000 [0.000, 0.000]. Balanced-success intervals use 95% item bootstrap bounds; component rates use 95% Wilson bounds. Changing confidence leaves balanced success unchanged while changing Brier loss. All 14 repeated conditions agree in their semantic traces. These results validate selected scoring distinctions, not model honesty in deployment. No learned model or human participant is evaluated.
Author: Pooria Arab
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7224414849