Reducing Faculty Burden with Question Creation: A randomized control trial comparing AI-generated versus expert-generated questions through student feedback and performance (Preprint)
Year: 2024
Type: preprint
Study designs: Randomized Controlled Trial, Clinical Trial
Abstract: BACKGROUND Developing high quality formative and summative assessment questions can take considerable time and effort, which in turn can contribute to burnout among medical school teaching faculty. Generative AI can decrease faculty burden by reducing the time needed to develop assessment materials. OBJECTIVE This study compares students’ performance on and subjective assessment of AI- versus expert-generated questions. We hypothesized that there would be no significant difference in student performance or subjective assessment of AI vs expert-generated questions. METHODS We designed a single-center, prospective, double-blind randomized controlled trial where student participants randomly received one AI or expert-generated question per day over the course of four weeks. RESULTS We found no significant difference between the distributions of the proportion of correct responses to AI- versus expert-generated questions (p=0.1809). Interestingly, the cumulative proportion of questions answered correctly over 28 days increased slightly for human-generated questions but stayed constant for AI-generated questions. There was a significant difference (p=0.0051) between the distribution of student difficulty ratings for AI- versus expert-generated questions. CONCLUSIONS Generative AI can reduce faculty burden by creating effective formative and summative assessments.
Authors: Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Syed
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
OpenAlex ID: W4404134320