Beyond Acuity Accuracy: A Multidimensional Framework for Evaluating Patient-Facing Emergency Triage AI Tools
Year: 2026
Type: preprint
Abstract: Patient-facing AI triage tools may support care-seeking and emergency department prioritization, but require evaluation beyond acuity accuracy alone. We developed SAFE-Triage, a framework integrating human review, automated metrics, LLM-as-judge evaluation, and simulation testing across clinical performance, patient interaction, and clinician-facing summary quality. We applied SAFE-Triage to 40 MIMIC-IV-derived emergency vignettes and a 10,030-case acuity analysis comparing prompted LLM triage (PLT) with rule-augmented prompted LLM triage (rPLT). PLT achieved higher emergency severity index (ESI) accuracy (57.5% vs 42.8%, $P < 0.001$), lower undertriage (13.1% vs 29.8%, $P < 0.001$), and higher clinician ratings across all domains. rPLT produced shorter and more stable outputs with higher automated readability scores. SAFE-Triage identified trade-offs between acuity accuracy, undertriage risk, communication quality, readability, and summary quality, supporting pre-deployment refinement and ongoing monitoring by showing where triage tools are efficient, need improvement, or need clinician oversight.
Source: arXiv (Cornell University)
Authors: Heerschap, Ruio, Foote, Henry, Alwakeel, Mahmoud, Qian, Zhouhan, Wang, Jiaqi, Bond, Elisabeth, Yang, Yang, Liew, Jonathan Chong Kai, Bergens, Matthew A., Arnold, Monique, Wang, Haoyuan, Yang, Rui, Yu, Kunyu, Paripati, Nikita R., Pimenta, Diana, Naimoli, Kyle, Wong, An-Kwok Ian, Economou-Zavlanos, Nicoleta, Vissoci, João Ricardo, Liu, Nan, Chowdhury, Anand, Hong, Chuan
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7236358397
