When the Confidence Score Doesn't Describe the Output: A Five-Month Audit of a Deployed Biomedical AI System
Year: 2026
Type: preprint
Abstract: AI systems for biomedical research increasingly report their own confidence: grounding ratios, citation grades, evidence scores. These self-reported metrics are almost never audited against the output they describe. We report a retrospective audit of 5,504 immutable, signed analysis records produced by a deployed biomedical hypothesis-generation system over five months (June to October 2026). In 87.3% of runs, the grounding value shown to users diverged from the system’s own trust-adjusted grounding value computed on the same record (mean 0.757 against 0.347). In 475 publicly reachable analyses the interface reported complete grounding while the trust-adjusted value was zero; these runs were not empty, carrying between one and four enriched pathways. The generating defect resolved in August 2026 through unrelated changes, but the published records continued to display the incorrect value until this audit. Over the same period the system’s own share-readiness flag was set on 0.56% of runs while the interface displayed complete grounding on 46.7%. A separate reconciliation of the same system’s cryptographic commitment ledger found a second, independent failure: a maintenance job deleted prediction records after they had been committed to Bitcoin-anchored Merkle roots, leaving 18 committed leaves across 3 roots permanently unresolvable on the system’s own public verification page. We argue this failure class is structural rather than incidental, propose a minimal audit protocol applicable to any system that publishes self-reported confidence, and release the audit queries. No stored records were altered.
Author: Idiakhoa, Oluwafemi
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7224998822