Minimal Audit Protocol for Published AI Reliability Figures in Deployed AI Systems (v0.2)
Year: 2026
Type: preprint
Abstract: AI systems increasingly publish figures about their own quality: evaluation scores, confidence levels, validation rates, grounding ratios. Readers rarely have a way to check them. This document is a procedure for doing so. It sets out ten steps, from reading the target's documents to contacting its author before publication; thirteen checks with explicit pass and finding criteria, and four supporting techniques; a rule for classifying each finding as a measurement, omission or framing problem; and a fixed report format. An applicability table says which checks to run for a given system, so the protocol can be applied to an evaluation harness, a retrieval-augmented agent or a deployed product. The protocol was first written for, and applied in, a self-audit of a deployed biomedical AI system (Idiakhoa, 2026), and revised after a second audit of an open-source retrieval-augmented agent. Thresholds marked as v0.2 defaults have not yet been validated across systems. The protocol is published so that a third party can apply it without either case study
Author: Idiakhoa, Oluwafemi
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225465117