Work

Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Year: 2026

Type: conference-paper

Abstract: Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors.As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential.We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution-feedback pairs and three feedback conditions 1 .Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most.These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits.Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness.Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue.

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2026.bea-1.56

Authors: Tahreem Yasir, Wenbo Li, Sam Gilson, Sutapa Tithi, Xiaoyi Tian, Tiffany Barnes

Institution: North Carolina State University

Language: English

Topic: Intelligent Tutoring Systems and Adaptive Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Funder: National Science Foundation