Work

Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System

Year: 2026

Type: conference-paper

Abstract: Intelligent Tutoring Systems (ITS) can record learner interactions in fine-grained detail at scale.This opens the door to data-driven methods for investigating system performance and identifying points for improvement.In this paper, we draw on authentic log data from an English language ITS (N logs = 5646, N students = 368) to investigate the performance of its feedback algorithm.In step 1 of our analysis, we profiled feedback accuracy by exploring how well the system provided errorspecific feedback to malformed student answers in gap-filling grammar exercises using an expert-created set of feedback generation rules.We then identified frequently occurring student errors that triggered incorrect or unspecific feedback and refined the rule set used to detect and respond to these errors with correct specific feedback.In step 2, we validated the rule modifications on an unseen dataset.Comparing the performance of the initial and updated rule sets, we find significant improvement that generalizes to unseen data.Our study thus illustrates how an empirical evaluation of authentic data can complement feedback creators' expertise by informing rule refinement decisions that yield significant and generalizable performance improvements to feedback in ITS systems.

Source: Workshop on Innovative Use of NLP for Building Educational Applications

DOI: 10.18653/v1/2026.bea-1.14

Authors: Mariia Soliar, Leona Colling, Stephen Bodnar, Detmar Meurers

Institutions: Leibniz-Institut für Wissensmedien, University of Tübingen

Language: English

Topic: Intelligent Tutoring Systems and Adaptive Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: hybrid

Funder: Bundesministerium für Bildung und Forschung