Reconceptualizing Scoring Reliability Through Linguistic Similarity
Year: 2025
Type: article
Abstract: Conventional cross-country scoring reliability in international large-scale assessments often depends on double scoring, which typically involves relatively small samples of multilingual responses. To extend the reach of reliability estimation, this study introduces the Linguistic-integrated Reliability Audit (LiRA), a novel method that measures scoring reliability using an entire dataset in a large-scale, multilingual context. LiRA automatically generates a second score for each response by analyzing its semantic alignment within a neighborhood of similar responses, then applies a weighted majority voting to determine a consensus score. Results demonstrate that LiRA provides a more comprehensive and systematic estimation of scoring reliability at the item, country, and language levels, while preserving the fundamental concepts of traditional reliability.
Institution: Boston College
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,650
OpenAlex ID: W4417491444