Work

Evaluating automated evaluation systems for spoken English proficiency: An exploratory comparative study with human raters

Year: 2025

Type: article

Abstract: Automated evaluation systems (AESs) for spoken language assessment are increasingly adopted in global educational settings, yet their validity in non-Western contexts remains underexplored. This study addresses this gap by examining three widely used Chinese-developed AES tools in their assessment of spoken English proficiency among 30 Chinese undergraduates. The study employed an IELTS-adapted speaking test, assessed simultaneously by AESs and human raters, with scoring alignment analyzed through intra-class correlation coefficients, Pearson correlations, and linear regression. Results revealed that two systems demonstrated strong agreement with human ratings, while the third exhibited systematic score inflation, likely due to algorithmic discrepancies and limited consideration of nuanced language features. Our findings suggest the potential of AESs as valuable complements to traditional language assessment methods, while highlighting the necessity for calibration and validation procedures. This research has significant implications for integrating AESs in educational contexts, particularly in English as a Foreign Language (EFL) settings, where they can enhance efficiency and standardization.

Source: PLoS ONE

DOI: 10.1371/journal.pone.0320811

Authors: Tianhui Chen, Sanjun Sun

Institutions: Suzhou City University, Beijing Foreign Studies University

Language: English

Topic: EFL/ESL Teaching and Learning

Subfield: Language and Linguistics

Field: Arts and Humanities

Domain: Social Sciences

SDG: Quality education

Open Access status: gold

Estimated APC paid: $2,382