Year: 2026
Type: preprint
Study design: Observational Study
Abstract: ABSTRACT BACKGROUND As artificial intelligence (AI) enters clinical practice, concerns about deskilling and never-skilling make the independent assessment of clinical competence increasingly important. The AI-Standardized Clinical Examination (ASCE) uses AI to simulate and score standardized clinical encounters, potentially overcoming some of the logistical and standardization constraints of conventional objective structured clinical examinations (OSCEs). Whether it provides valid and reliable assessment at scale is unknown. METHODS In this prospective multicenter cohort study, final-year medical students from six French medical schools completed a 10-station ASCE six weeks before the national OSCE session. GPT-4o simulated standardized clinical partners; transcripts were subsequently scored by GPT-4.1 against faculty-defined rubrics. The primary outcome was the correlation between ASCE and national OSCE scores. Secondary outcomes were inter-rater reliability of expert transcript scoring, alignment of ASCE scores with expert ratings, and internal consistency. RESULTS Among 948 enrolled students, 588 had paired ASCE and national OSCE scores. ASCE scores were correlated with national OSCE performance (Spearman’s ρ, 0.57; 95% CI, 0.51 to 0.63), with less than 1% of residual variance attributable to center (intraclass correlation coefficient, 0.01). Eight experts scored 20 randomly selected ASCE transcripts; Krippendorff’s α was 0.77 overall, 0.80 for clinical skills, and 0.70 for communication and attitude skills. Adding GPT-4.1 as an evaluator reduced α by 0.02, and its scores did not differ significantly from those of individual experts. Internal consistency was 0.67 across 10 stations and was projected to approach 0.80 with 20 stations. CONCLUSIONS Scores from an examination delivered and scored by AI were associated with performance on a national high-stakes OSCE, with minimal center-level variation, tentative agreement among expert raters, and automated scores that aligned with expert ratings. These results support AI delivery and scoring of clinical examinations, with expert review of automated scores. Description In what is, to our knowledge, the first prospective multicenter study to evaluate a clinical examination delivered and scored by AI against an independent national OSCE, 588 final-year medical students across six French medical schools completed an AI-Standardized Clinical Examination before the independent national OSCE. Scores correlated with national OSCE performance (Spearman’s ρ, 0.57), with negligible variation between centers, supporting AI-standardized assessment at scale.
Source: medRxiv
Authors: Antonio Lopez, Ludovic Gabellier, Clément Peytavi, Zeno Loi, Emma Poupin-Lavigne, Catherine Plotton, Vincent Dupont, Léon Jahdauti, Philine de Vries, Philippe Thuillier, Maxence Compagnat, Pascale Vergne-Salle, Mathieu Kuchenbuch, Eva Feigerlova, Philippe Berthelot, Nathalie Bednarek, Béatrice Cochener-Lamard, Pierre-Yves Robert, Marc Braun, Denis Morin, Philippe Deruelle, Isabelle Laffont, Yves-Marie Pers, Kévin Yauy
Institutions: Université de Montpellier, Centre Hospitalier Universitaire de Montpellier, Lyon 1 Université, Parcours santé systémique, Université Jean Monnet, Université Jean Monnet, Centre Hospitalier Universitaire de Reims, Université de Reims Champagne-Ardenne, Université de Bretagne Occidentale, Centre Hospitalier Régional Universitaire de Brest, Inserm, Centre Hospitalier Universitaire de Limoges, Handicap, Activité, Vieillissement, Autonomie, Environnement, Université de Limoges, Contrôle de l’Activation cellulaire, Progression Tumorale et Résistance thérapeutique, Centre National de la Recherche Scientifique, Centre Hospitalier Régional et Universitaire de Nancy, Ingénierie Moléculaire, Cellulaire et Physiopathologie, Université de Lorraine, Nutrition-Génétique et Exposition aux Risques Environnementaux, École Normale Supérieure de Lyon, Centre Hospitalier Universitaire de Saint-Étienne, Centre International de Recherche en Infectiologie, Centre de Recherche en Sciences et Technologies de l'Information et de la Communication, Laboratoire de Traitement de l'Information Médicale, Anti-infectieux : supports moléculaires des résistances et innovations thérapeutiques, Imagerie Adaptative Diagnostique et Interventionnelle, IMT Mines Alès, EuroMov Digital Health in Motion, Institute for Regenerative Medicine & Biotherapy, Hôpital Lapeyronie, Laboratoire d'Informatique, de Robotique et de Microélectronique de Montpellier
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7215907590