Year: 2025
Type: conference-paper
Abstract: While the Question Generation (QG) task has been increasingly adopted in educational assessments, its evaluation remains limited by approaches that lack a clear connection to the educational values of test items.In this work, we introduce test item analysis, a method frequently used by educators to assess test question quality, into QG evaluation.Specifically, we construct pairs of candidate questions that differ in quality across dimensions such as topic coverage, item difficulty, item discrimination, and distractor efficiency.We then examine whether existing QG evaluation approaches can effectively distinguish these differences.Our findings reveal significant shortcomings in these approaches with respect to accurately assessing test item quality in relation to student performance.To address this gap, we propose a novel QG evaluation framework, QG-SMS, which leverages Large Language Model for Student Modeling and Simulation to perform test item analysis.As demonstrated in our extensive experiments and human evaluation study, the additional perspectives introduced by the simulated student profiles lead to a more effective and robust assessment of test items.
Authors: Bang Nguyen, Tingting Du, Mengxia Yu, Lawrence Angrave, Meng Jiang
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
Awards: RI: Small: Learning to Retrieve Structured Information for Summarization and Translation of Unstructured Text, SHF: Medium: Collaborative Research: Semantically-Enhanced Software Traceability for Supporting Human-Centric Tasks, CAREER: Synergistic Approaches for Specialized Intelligent Assistance, G4720003262, G495881888, Collaborative Research: Advancing STEM Online Learning by Augmenting Accessibility with Explanatory Captions and AI, III: Small: Intelligent Scientific Text Analytics with Knowledge-Augmented Abductive Reasoning, G8876996369
OpenAlex ID: W4412889909