Evaluating the Quality of AI-Written Scenarios for Virtual Oral Surgical Board Preparatory Examination
Year: 2025
Type: article
Abstract: • AI-written scenarios were used in a virtual oral board preparatory examination (mock orals) held by Washington University in St. Louis and Saint Louis University. • Faculty rated AI-written scenarios as level-appropriate for senior residents. • Residents found AI-written scenarios equally challenging as faculty-written scenarios. • Faculty and residents could not reliably identify the source of scenarios. • AI-written scenarios may augment oral board preparation in surgery. The objective of this study was to evaluate whether ChatGPT can generate level-appropriate clinical scenarios that are suitable for use in oral board preparatory examination (mock oral exam) for senior surgical residents. This was a prospective, blinded study in which AI-written and faculty-written scenarios were reviewed, randomized and used for testing in virtual mock oral exam. Both faculty examiners and test-taking residents were blinded to the true authorship of the scenarios. After the examination, participants completed a survey evaluating the complexity of each scenario and their perceptions of its authorship. The study was conducted at Washington University in St. Louis (WashU), an academic medical center located in St. Louis, Missouri. The participating institutions also included Saint Louis University (SLU). Study participants included twenty-five senior general surgery residents (PGY4 and PGY5) and twenty faculty examiners from WashU and SLU, who took part in virtual mock oral examination. Post-exam surveys were completed both residents and faculty. Faculty rated most AI-written and faculty written scenarios as "level-appropriate” in terms of both the quality of the text and the degree of complexity. Similarly, when residents were asked to identify the most difficult scenarios, they selected both AI- and faculty-written scenarios at comparable rates. Notably, both faculty and residents struggled to correctly distinguish the origin of the scenarios, with frequent misidentification across both groups. AI-written clinical scenarios were comparable to faculty-written scenarios in terms of complexity and appropriateness for senior surgical residents when used in a virtual mock oral board examination, highlighting the potential utility of AI-based tools in oral board preparation and surgical education.
Source: Journal of surgical education
Authors: Usman Y. Panni, Christa Donald, Jeffrey A. Blatnik, Michael Williams, Jennifer Yu, Paul E. Wise
Institutions: Washington University in St. Louis, Saint Louis University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $3,420
OpenAlex ID: W4415428409