Artificial intelligence as a tool for formative OSCE feedback: A mixed-methods analysis of student perceptions and feedback characteristics
Year: 2026
Type: article
Abstract: Introduction: The Objective Standardized Clinical Examination (OSCE) is widely used to assess and provide feedback to healthcare students; however, feedback quality remains a common source of frustration, with literature indicating that students feel feedback is untimely, insufficiently comprehensive, and/or not individualized. Objective: In this prospective, mixed-methods study, we use a custom artificial intelligence (AI) model – the Communication and Learning Improvement Model for Bedside Skills (CLIMBS) – to generate preliminary feedback for medical students following OSCE performances. Our primary aim is to compare AI-generated quantitative feedback with Standardized Patient (SP) evaluations, comparing accuracy and assessing student perceptions of feedback quality. Methods: Using CLIMBS, unedited transcripts of 15 first-year medical student OSCE encounters were analyzed using a 17-metric communication and data gathering rubric, divided into two categories: subjective and semi-objective. SP- and CLIMBS-generated scores were compared on an overall and categorical basis, and the impacts of transcript accuracy was assessed; qualitative comparison is ongoing. Faculty review for objective score confirmation is ongoing, review by a researcher used in the interim. Student perceptions of feedback were elicited via free-response and 10-point Likert surveys. Results: Overall scores assigned by SPs and CLIMBS were significantly different (SP=95.00±6.97 vs CLIMBS=81.93±3.96; p<0.0001), with SPs reporting more-lenient scores. Similar differences were seen in subjective (SP=96.67±12.91 vs CLIMBS=78.40±7.40; p=0.0002) and semi-objective subscores (SP=94.00±5.07 vs CLIMBS=84.67±6.94; p<0.0001). CLIMBS exhibited substantial reliability with human reviewer (Cohen Kappa=0.699) Medical students evaluated CLIMBS feedback as more comprehensive (CLIMBS=8.083±1.311 vs SP=5.667±2.060; p=0.0071) and more actionable (CLIMBS=8.308±1.182 vs SP=7.154±1.725; p=0.0411), but reported no difference in perceived accuracy between modalities (CLIMBS=7.692±0.9473 vs SP=7.923±1.656; p= 0.4890). Conclusion: CLIMBS-generated OSCE feedback was more stringent than Standardized Patient scoring, yet was perceived by students as similarly accurate, and more comprehensive and actionable. These findings suggest AI-assisted feedback may complement traditional evaluation methods by adding discriminative detail. Ongoing faculty review will further define validity and educational impact.
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7155754353
