Work

Can LLMs Reliably Simulate Real Students’ Abilities in Mathematics and Reading Comprehension?

Year: 2025

Type: conference-paper

Abstract: Large Language Models (LLMs) are increasingly used as proxy students in the development of Intelligent Tutoring Systems (ITSs) and in piloting test questions.However, to what extent these proxy students accurately emulate the behavior and characteristics of real students remains an open question.To investigate this, we collected a dataset of 489 items from the National Assessment of Educational Progress (NAEP), covering mathematics and reading comprehension in grades 4, 8, and 12.We then apply an Item Response Theory (IRT) model to position 11 diverse and state-of-the-art LLMs on the same ability scale as real student populations.Our findings reveal that, without guidance, strong general-purpose models consistently outperform the average student at every grade, while weaker or domain-mismatched models may align incidentally.Using gradeenforcement prompts changes models' performance, but whether they align with the average grade-level student remains highly model-and prompt-specific: no evaluated model-prompt pair fits the bill across subjects and grades, underscoring the need for new training and evaluation strategies.We conclude by providing guidelines for the selection of viable proxies based on our findings.1

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2025.bea-1.75

Authors: KV Aditya Srivatsa, Kaushal Maurya, Ekaterina Kochmar

Institution: Mohamed bin Zayed University of Artificial Intelligence

Language: English

Topic: Intelligent Tutoring Systems and Adaptive Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

OpenAlex ID: W4412889178