Simulated student fidelity: a preregistered comparison of model-generated and real student turns on five measures in held-out classroom transcripts
Year: 2026
Type: preprint
Abstract: Tools that let teachers rehearse a lesson put a language model in the student's chair. Whether the model talks like a child can be measured before anyone asks whether the rehearsal helps. This preregistered study generated student turns for 255 upper-elementary mathematics lessons from the NCTE transcripts, holding each teacher's real turns fixed, and compared the generated turns with the real ones on five measures: utterance length, the share of short answers, lexical richness, question rate, and confusion markers. Each measure had a fixed equivalence bound set from the spread between real classrooms. Two arms differed only in the system prompt: a plain instruction, and a prompt describing a student archetype written after the exploratory phase. The generator was Qwen2.5-32B-Instruct, open weights pinned by hash, after the registered commercial model failed on most turns. Real student turns, one utterance each in the transcript file, averaged 4.6 words, and 60% were three words or fewer. The plain arm's replies averaged 11.1 words and 8.0% were that short; the archetype arm's 29.2 words and 1.3%. Both arms asked fewer questions than real students and used fewer confusion words. One of ten arm-by-measure tests reached equivalence: lexical richness in the archetype arm. On the calibration hypothesis, the archetype prompt moved the model farther from real students on four of five measures and closer on vocabulary, each Holm-significant at α = 0.05 in the primary analysis; the registration had predicted improvement on two of the four. A registered contamination rule was triggered by three exact matches, of one, one and five words, in the canary probes, which made the analysis excluding five flagged teachers primary; every fidelity verdict is the same with those teachers kept, and two of the five calibration tests lose significance. The paper also records how the generation step failed three times before it was gated.
Source: EdArXiv (OSF Preprints)
Author: Kumar Srivastava
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7225468062