A Warning Number for Rule-Checked AI: Seeing When the Rules Stop Covering the Questions
Year: 2026
Type: article
Abstract: Formal verifiers such as Lean are used to check AI answers against rules written by experts for a field, for example tax, law or drug discovery. The verifier checks the answer against the rules, but it cannot check the rules themselves. If a rule goes out of date, an answer can pass the check and still be wrong, and such a problem is usually found late, through a complaint, an audit or the news. This concept note describes one number, the settling number, calculated from the verifier's own stream of results: for each question, checked or could not be checked. When the rules cover fewer of the questions, the recovery slows and the number rises. It gives a warning while the answers are still passing, and it works for changes that nobody announced. Each level of the number has a defined action, decided by the computer: Steady, Watch, Look closer. At the highest level the questions of that type are held for the experts who write the rules, and the answers shown are labelled as not checked. A person can review the work at any state. The limits between the states are withheld until they are checked on real streams. Today's systems: the verifier checks each answer, and experts update the rules by following changes in the law and scheduled reviews.Our method: the settling number of the verifier's own stream shows that the rules are covering fewer questions, with a defined action at each level. Patent pending.
Author: Vaithyanathan, Prakash
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7223050207