Work

How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study

Year: 2026
Type: article
Abstract: Grading physical-examination performance in objective structured clinical examinations (OSCEs) is labor-intensive, difficult to scale, and prone to inter-rater variability. In this retrospective ablation study of 300 OSCE encounters from six stations at the UT Southwestern Simulation Center, we examined how the agreement of a multimodal AI system (Gemini 2.5 Pro) with a composite reference standard changed as recording inputs were removed, and compared it with that of routine standard-of-care grading by non-physician standardized-patient evaluators (SPEs). Agreement was measured with quadratic-weighted kappa, and configurations were compared using paired learner-cluster bootstrap tests with false-discovery-rate correction. Against a composite reference standard with joint adjudication of discordant items by a physician and an emergency medical technician (EMT), three-camera native video achieved the highest agreement (quadratic-weighted κ = 0.832; 95% CI 0.775–0.885). Across isolated-modality configurations, agreement point estimates were ordered native video > audio-only > transcript-only > visual-only; the audio-only versus transcript-only difference was not statistically significant (raw p = 0.29; adjusted p = 0.36). Without audio, agreement dropped substantially and the system did not reliably distinguish performed from partially performed or omitted maneuvers (three-camera silent video κ = 0.239, 95% CI 0.167–0.307; full credit for 73.9% of items the reference scored below full credit). Agreement therefore depended largely on audio-inclusive inputs, which in OSCEs carry students’ verbal signposting and standardized-patient responses. Three-camera native video also agreed more closely with the reference than routine SPE grading (κ = 0.731; 95% CI 0.692–0.763; paired difference 0.10, 95% CI 0.04–0.16; unadjusted p = 0.003). For back range of motion, however, routine grading awarded full credit by instruction, whereas the reference applied the written rubric; without that item the difference was not statistically significant ( p = 0.17). These findings map recording requirements and limitations of AI-assisted physical-examination scoring in structured OSCEs. The locally constructed composite reference used non-physician SPE and EMT ratings; a physician and the EMT jointly adjudicated only discordant items, so the reference does not independently establish physical-examination competence. Generalization to unscripted clinical encounters remains to be tested.
Language: English
Field: Medicine
Open Access status: gold
Estimated APC paid: $2,990
OpenAlex ID: W7226019716
FWCI:
Cites:
Cited by:
Related to: 10