Work

Advancing Language and Literacy Assessments through Technological Integrations

Year: 2024
Type: dissertation
Abstract: Technological advances are reshaping language and literacy assessment practices and the changes are accompanied by increased calls for research on validity, reliability, and fairness. This dissertation comprises three empirical studies that respond to the call by taking multi-faceted inquiry approaches. The first study investigated how well natural language processing (NLP)-derived linguistic and acoustic features predicted human oral language proficiency (OLP) scores, using five proficiency rating criteria and holistic theta scores derived from item response theory. Findings aligned with previous research and revealed moderate machine accuracy when predicting analytic human scores for semi-structured speech. The study also discussed the challenges of evaluating grammar independently from other OLP components and called for further research into the efficacy of holistic scales over analytic scales for high-stakes testing. Study 2 utilized many-facet Rasch measurements to examine differences in human and machine rating behaviours, testing invariance across students’ language learner status, task type, and OLP analytic criteria. While human and machine raters applied the OLP scale consistently, unique biases were observed from machine and L1 human raters relating to task type, grammar, and vocabulary. This study advocated for a greater integration of psychometric approaches like MFRM with machine learning model performance metrics to identify and address specific biases, noting that conventional metrics like F1 scores might not fully reveal nuanced biases. The final study employed cognitive diagnostic modelling, dynamic assessment, and automated OLP scores to examine the impact of language comprehension on reading comprehension skill mastery within school and community settings. Pairing diagnostic assessment with human and machine mediation benefited L2 students. Results supported the simple view of reading, highlighting the significant role of decoding ability in differentiating students’ comprehension levels. Findings also provided substantial evidence that the combination of real-time diagnostic feedback with dynamic human mediation had a transformative impact on students' metacognitive processes, self-regulated learning, learning beliefs, and behaviours. These studies demonstrated artificial intelligence's (AI) complementary role in supporting human expertise in educational assessments. The human-centred loop ensures that AI assessments support student growth and development rather than being confined to evaluative or decision-making purposes. It shows that AI is not replacing traditional methods but enhancing them, combining the best of both worlds to better serve educational needs.
Language: English
Open Access status: green
OpenAlex ID: W7225767793
Cites: 308
Cited by:
Related to: 10