Work

Generative AI for OSCE and Workplace-Based Assessment: Mapping Validity, Bias/Fairness, Reliability, and Defensible Governance

Year: 2025
Type: article
Study design: Systematic Review
Abstract: Background: Generative artificial intelligence (GenAI), including large language models (LLMs), is increasingly used in medical education assessment as item writers, rubric-based scorers of constructed responses, and draft generators for narrative feedback. In high-stakes settings (e.g., OSCE pass/fail and programmatic workplace-based assessment [WBA]), GenAI functions as a measurement intervention and therefore requires defensible evidence for validity, fairness, reliability, and governance. [1,2,11] Methods: We conducted a PRISMA-ScR-aligned scoping review (2020-present) across major medical education and health databases to map empirical and review evidence on GenAI in assessment, prioritizing OSCE/WBA and high-stakes contexts. Evidence was charted using Kane’s argument-based validity framework (scoring, generalization, extrapolation, implications), alongside themes related to bias/fairness, reliability, and governance. [9-11] Results: Evidence is heterogeneous and concentrated in written assessment (short answers, clinical notes, and post-OSCE documentation) and in automated analysis of narrative WBA feedback. LLM grading can be efficient, but studies show variable agreement with expert human ratings and potential shifts in grade distributions. In early OSCE-related studies, agreement between GenAI outputs and clinician judgments is inconsistent, highlighting threats to the scoring inference and the need for response-process evidence, rater calibration, and drift monitoring. [3,6,16-21] Conclusion: For clinical skills and high-stakes decisions, GenAI use should be bounded and treated as a regulated assessment intervention. Institutions should require a complete validity argument, explicit fairness audits, program-level reliability evidence, and governance controls (versioning, audit trails, appeals, and change management). We propose a practical DEFEND-VFRG toolkit (Validity, Fairness, Reliability, Governance) for implementation and reporting. [11-15,25]
Language: English
Field: Medicine
Open Access status: diamond
OpenAlex ID: W7217859055
FWCI: 0.147
Cites: 21
Cited by: 1
Related to: 10