Work

Helping a Weak AI Checker: More Catches, Many More False Alarms

Year: 2026
Type: preprint
Abstract: Doctors now ask AI tools what to do about a patient, and they rarely have time to check each answer against the guidelines before acting on it. One safeguard is a second AI model, the checker, which reads the first one's answer and says whether it is safe. To save money and time, the checker is often a smaller and less capable model than the one it checks. It can be wrong in two ways: it can pass an unsafe answer (a miss), or it can flag a safe one (a false alarm). We asked whether a weak checker does better when it is given help. We wrote 48 clinical questions of the kind a GP might ask. Each had one safe answer and two unsafe answers, and each unsafe answer was the safe one with a single error deliberately put in. That made 144 answers. The checker judged every answer in four ways (Figure 3). In direct checking, the baseline, it reads the question and the answer and says safe or unsafe. In fact-by-fact checking, it lists the facts a safe answer must contain before it sees the answer, and then checks the answer against its list. In single-call checking, it writes the list and gives its verdict in one pass, with the answer in view from the start. In checking with critique, a stronger model writes notes on the answer, and the checker reads them before giving its verdict. Each way was run with and without a short summary of the relevant UK guidance placed in front of the checker. Three AI models took the checker's role. The main one was Claude Haiku 5.5, a small and cheap model, and the stronger model that wrote the notes was Claude Opus 5.5. We measured two things each time: the share of unsafe answers that the checker flagged (its catch rate) and the share of safe answers that it wrongly flagged (its false-alarm rate). None of the three kinds of help clearly raised the catch rate without also raising the false-alarm rate. With fact-by-fact checking, the main checker caught 98.4% of unsafe answers, up from 88.5% with direct checking, but it also wrongly flagged 68.1% of safe answers, up from 11.8%. Before the experiment we had decided that fact-by-fact checking would count as an improvement only if false alarms rose by less than 10 percentage points. They rose by far more, so by our own test it failed. A checker that flags most safe answers tells the doctor reading it very little. Most of the extra catches were of one kind of error, where something important had been left out of the answer. The cheapest help was the guidance summary, which lowered false alarms for two of the three checkers without a clear change in catch rate. The stronger model's notes passed its mistakes on to the checker, mostly in one direction. The stronger model also judged every answer itself, and it made 26 false alarms and 56 misses. Of the 26 false alarms, the main checker repeated 11 when it checked directly and 23 after it had read the notes. Of the 56 misses, it repeated 45 when it checked directly and 14 after it had read the notes. All of these results rest on one doctor's judgement. One of us (KY) decided alone which answers were safe and which were unsafe. The other (OE) then graded a sample of 49 answers without seeing KY's decisions and disagreed on 16, each time calling safe an answer that KY had called unsafe. On the 33 answers we agreed on, the rise in false alarms is still there, but the rise in catches is not confirmed.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7236632345
Cites:
Cited by:
Related to: 9