Work

Retrospective testing and prospective randomized trial of guardrails and a verifier for AI care plans

Year: 2026
Type: preprint
Study design: Clinical Trial
Abstract: Health systems are adding verifiers, AI models that check another AI’s drafted care plans, to reduce errors and omissions. We tested a verifier in two ways. First, three AI models drafted care plans for 447 retrospective test cases; a verifier caught 5 of 17 errors checkable against the patient’s record when counting flags for wrong recommendations, and 10 of 17 when also including flags for missing care. Second, in a prospective trial (NCT07718893), we randomized 240 patients in complex primary care management programs in four states to unassisted physicians, physicians with a guardrailed AI assistant, or physicians with the guardrailed assistant plus a verifier. The verifier raised 11 alerts across 77 encounters; physician adjudicators judged 9 correct, but clinicians in the trial acted on only 1. Safety events were similar across arms. Effective verification may therefore require alerts that clinicians must resolve, despite the risk of alert fatigue.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7235569563
Cites: 38
Cited by:
Related to: 10