Work

Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study

Year: 2026
Type: article
Abstract: BACKGROUND: Health systems are increasingly deploying patient-facing conversational AI platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement. OBJECTIVE: This study aimed to evaluate a structured red teaming methodology for the safety assessment of a patient-facing clinical AI platform and determine whether targeted remediation guided by initial testing reduced safety failures in subsequent rounds of high-risk evaluations. METHODS: A 2-phase prospective simulation study was conducted at a single academic health system in January and February 2026. Phase 1 enrolled 48 health care professionals who completed 6 randomized clinical scenarios (3 medium risk and 3 high risk). Following independent platform remediation informed by aggregated phase 1 findings, phase 2 enrolled 27 additional participants who completed 8 targeted high-risk scenarios. All 476 transcripts were independently scored by 3 calibrated raters using a 5-domain safety rubric. A strict firewall separated the research team from the platform developer, and only aggregated, deidentified findings were shared between phases. RESULTS: In phase 1, the high-risk failure rate was 30.2% (35/116) of transcripts vs 5.6% (7/126) for medium-risk scenarios. In phase 2, the high-risk failure rate was 8.5% (20/234), representing an absolute difference of 21.7 percentage points (participant-clustered 95% CI 11.1-32.2) and a 72% relative difference, with comparable performance across prespecified and improvised prompts within the targeted failure categories. Interrater reliability was moderate to strong in phase 1 (intraclass correlation coefficient=0.66, 95% CI 0.62-0.69) and moderate in phase 2 (intraclass correlation coefficient=0.49, 95% CI 0.43-0.55). Agreement on the binary failure classification remained stable between phases (Fleiss κ=0.70 in both phases), with observed agreement increasing from 90.9% (220/242) to 94.9% (222/234). Residual failures clustered around triage calibration errors and scope violations. CONCLUSIONS: Structured iterative red teaming was associated with improved safety performance on new high-risk scenarios following targeted remediation. The 5-domain rubric and scenario taxonomy offer a replicable methodology for health systems conducting predeployment and ongoing clinical AI safety assessment.
Source: JMIR AI
Institution: Hartford Hospital
Language: English
Field: Medicine
Open Access status: diamond
OpenAlex ID: W7143412752
FWCI:
Cites: 23
Cited by:
Related to: 10