Work

Toward Psychometric Learning Analytics: Augmenting the Urnings Algorithm with Response Times

Year: 2025

Type: article

Abstract: Adaptive learning systems (ALS) aim to tailor the educational material to match the student's needs, ultimately improving the learning outcomes. An ALS dynamically adjust the level of practice based on the student's ability; therefore, obtaining accurate ability estimates is crucial. Since the amount of responses in a timeframe is limited, high measurement precision is unattainable using only accuracies, which calls for the inclusion of other data sources into the measurement. Here, we propose algorithms that can estimate the abilities on‐the‐fly based on both accuracy and response times (RT). These are extensions of a rating system called the Urnings algorithm. Since the Urnings algorithm uses discrete updates, building on the difference between a single observed and simulated response, we combined accuracy and RT into a continuous score using a discretized version of the Signed Residual Time (SRT) scoring rule. Through simulation studies, we showed that by augmenting the algorithm with RT, a reliable ability measure and better ability tracking can be obtained by administering fewer items. By reanalyzing data from an existing ALS, we showed that the algorithms can be utilized even if the SRT scoring rule is not explicitly used during measurement, providing better ability estimates and smaller prediction errors.

Source: Journal of Educational Measurement

DOI: 10.1111/jedm.70026

Authors: Bence Gergely, Maria Bolsinova

Institutions: Tilburg University, Károli Gáspár University of the Reformed Church in Hungary

Language: English

Topic: Psychometric Methodologies and Testing

Subfield: Management Science and Operations Research

Field: Decision Sciences

Domain: Social Sciences

SDG: Quality education

Open Access status: hybrid

Estimated APC paid: $3,040