Year: 2016
Type: article
Abstract: The e‐rater® automated scoring engine used at Educational Testing Service (ETS) scores the writing quality of essays. In the current practice, e‐rater scores are generated via a multiple linear regression (MLR) model as a linear combination of various features evaluated for each essay and human scores as the outcome variable. This study evaluates alternative scoring models based on several additional machine learning algorithms, including support vector machines (SVM), random forests (RF), and k‐nearest neighbor regression (k‐NN). The results suggest that models based on the SVM algorithm outperform MLR models in predicting human scores. Specifically, SVM‐based models yielded the highest agreement between human and e‐rater scores. Furthermore, compared with MLR, SVM‐based models improved the agreement between human and e‐rater scores at the ends of the score scale. In addition, the high correlation between SVM‐based e‐rater scores with external measures such as examinee's scores on the other parts of the test provided some validity evidence for SVM‐based e‐rater scores. Future research is encouraged to explore the generalizability of these findings.
Source: ETS Research Report Series
DOI: 10.1002/ets2.12094
Institution: Educational Testing Service
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: bronze
OpenAlex ID: W2295056205
