A 70 Clip Trap Set for Measuring Invented Drug Names in Veterinary AI Scribes
Year: 2026
Type: preprint
Abstract: AI scribes for veterinary medicine typically pair a speech recognition layer with a language model layer that drafts the clinical note. Either layer, or the two together, can add a medication to the record that nobody said. We built a blind, reproducible 70 clip test set, 45 ordinary dictations and 25 traps designed to bait a false drug name, written by an author with no access to the underlying matching or scoring code. We ran it once, on the same day, against a commercial veterinary AI scribe on a trial account with its default settings, and against our own scribe, Perch, with its out of the box lexicon. Eighty four drug mentions were dictated across the 45 dictations. We report raw counts, not rates: mentions found, mentions missed, false offers, and cases where one half of a dictated brand and generic pair was dropped. We describe every case where a product charted a drug that nobody said, including one from our own product's development history, and one case where the other product recovered a drug that our own product missed on this run. The full clip set, scoring script, and per clip ledger are published so anyone can rerun the test.
Authors: Usama, Mohammed, Nisar, Mustafa
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7225558687