Can Language Models Grade Algebra Worked Solutions? Evaluating LLM-Based Autograders Against Human Grading
Year: 2025
Type: conference-paper
Abstract: In this study, we investigate the feasibility of using LLMs as autograders by evaluating multiple models—OpenAI's GPT-4o, GPT-4.1-mini, and GPT-4.1-nano—on a manually graded open-source dataset of college algebra worked solution responses. For each model, we assess performance both with and without a self-consistency grading approach. We compare LLM-generated correctness labels to human annotations across 18,000 responses. Results show that GPT-4.1-mini achieves the highest accuracy (94.47\% with self-consistency), followed by GPT-4.1-nano (93.07\%) and GPT-4o (91.93\%). These findings suggest that self-consistency can slightly improve grading reliability, and that even compact models like GPT-4.1-mini and GPT-4.1-nano can approach human-level agreement in algebra autograding.
Authors: Shreya Bhandari, Zach Pardos
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W6930735756
FWCI:
Cites:
Cited by:
Related to: