Work

Evaluations of Automated Scoring Systems in Practice

Year: 2020
Type: article
Abstract: This research report provides a description of the processes of evaluating the “deployability” of automated scoring (AS) systems from the perspective of large‐scale educational assessments in operational settings. It discusses a comprehensive psychometric evaluation that entails analyses that take into consideration the specific purpose of AS, the test design, the quality of human scores, the data collection design needed to train and evaluate the AS model, and the application of statistics and evaluation criteria. Finally, it notes that an effective evaluation of an AS system requires professional judgment coupled with statistical and psychometric knowledge and understanding of the risk assessment and business metrics.
Authors: Ourania Rotou, André A. Rupp
Language: English
Open Access status: bronze
OpenAlex ID: W3029540124
FWCI: 0.405
Cites: 24
Cited by: 13
Related to: 9