SEXTANT: Measuring Six-Nines Residual Error for LLM Decisions Behind Independent Deterministic Checks
Year: 2026
Type: preprint
Abstract: No language model is accurate to six nines, yet process automation is often described with such figures. We study a narrower quantity that can be measured: the residual error of one model step wrapped in deterministic checks, defined as the probability that an output is wrong and accepted. On synthetic payment documents with truth known by construction, one model (Claude Haiku 5.5) was wrong on about 9% of documents. We report four results. First, the cheap way to certify a low residual, multiplying the model's error rate by the checks' miss rate on injected errors, failed: it bounded the residual at 9.19 × 10−4 while a direct count on 300,000 documents found 4.91 × 10−3, 5.3 times higher. The model's errors were not like injected errors. Every wrong output that passed the checks was of one kind, located exactly where the checks could not see. Second, one further deterministic rule closed that gap, and a direct zero-failure test on fresh documents found no wrong accepted output in 300,000 and then in 3,000,000 documents, a one-sided 95% upper bound of 9.99 × 10−7 per output processed for the 2,000 vendors of the test world. Because errors cluster by vendor, that bound does not carry over to new vendors (about 2 × 10−5 for a new set). A repeat with the same model through the provider's interface, on a fresh world of 50,000 vendors with about 60 documents each again found no wrong accepted output in 3,000,000 documents (bound 9.99 × 10−7), and 1.78 × 10−6 for a new set of vendors from the same generator. The price was automation: the share of outputs accepted without review fell from 91.6% to 69.7% on the 2,000-vendor world. Third, the same checks ported to a FlowX integration workflow, run on the FlowX platform in a demo (non-production) environment with a second model (GPT-6 Luna), gave verdicts identical to the reference implementation on every one of 2,999,805 documents run end to end, and accepted 0 wrong outputs among them, a one-sided 95% bound of 9.99 × 10−7 per output processed on the 2,000-vendor world. A second pre-registered platform count, on the 50,000-vendor world, accepted no wrong output in 599,952 documents, a bound of 5.58 × 10−6 for a new set of vendors: five nines, not six. Outputs the checks could not verify were not accepted; the use case decides whether such an output is rejected, retried or sent to a person. They were the model's errors (about 8%) and correct outputs the ledger could not tell apart from another invoice (about 22% on this deliberately hard synthetic mix). Fourth, on 387 real invoices most of the model's errors (69%) were of a kind injected errors never produce. The bounds below 10−4 concern one synthetic task. They support no statement about production systems or real invoices, and the real-invoice results are percent-level. What they do support is a method and a warning: a residual error rate is a property of a model, a prompt, a set of checks and a document mix together, and it has to be counted, not composed. Laboratory study on synthetic documents. Not a production measurement.
Author: Răduță, Bogdan
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: residual error, process automation
Open Access status: green
OpenAlex ID: W7236023353
