Year: 2026
Type: review
Study designs: Systematic Review, Meta-Analysis
Abstract: Despite a burgeoning body of research on using AI scoring systems in English assessments, concerns regarding their reliability persist. To fill this gap, this meta-analysis examined the AI-human scoring differences and the variables moderating these differences by synthesizing the results of 21 empirical studies with a total of 401,698 participants. Results indicate no statistically significant differences between AI and human scoring; the small effect size implies that the average systematic difference between the two was relatively modest. However, extremely high heterogeneity suggests that this overall finding masks considerable variability across study conditions. Moderator analyses reveal that AI-human scoring differences are significantly influenced by factors such as AI system type, number of human raters, agreement index employed, learner proficiency level (CEFR), and publication year. These findings suggest that while AI cannot fully replace human judgment, it can serve as a diagnostic reference tool within broader quality assurance frameworks. When significant discrepancies arise, they warrant investigation of both scoring sources. Based on these findings, this study offers evidence-based recommendations for educators on the effective use of AI scoring systems in language assessments.
Source: Scientific Reports
Authors: Bingwei Li, Xiao Qunhan, Chunhua Mao
Institutions: Hunan Normal University, Hunan Vocational College of Commerce
Language: English
Subfield: Language and Linguistics
Field: Arts and Humanities
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7154333889