Year: 2025
Type: conference-paper
Abstract: Advances in Large Language Models (LLMs) have sparked growing interest in their potential as explainable text evaluators.While LLMs have shown promise in assessing machinegenerated texts in tasks such as summarization and machine translation, their effectiveness in evaluating human-written content-such as student writing in classroom settings-remains underexplored.In this paper, we investigate LLM-based specificity assessment of student reflections written in response to prompts, using three instruction-tuned models.Our findings indicate that although LLMs may underperform compared to simpler supervised baselines in terms of scoring accuracy, they offer a valuable interpretability advantage.Specifically, LLMs can generate explanations that are faithful, non-repetitive, and exhibit high fidelity with their input, suggesting potential for enhancing the transparency and usability of automated specificity scoring systems.
Authors: Mohamed Elaraby, Diane Litman
Institution: University of Pittsburgh
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
Awards: Collaborative Research: EAGER: Developing and Optimizing Reflection-Informed STEM Learning and Instruction by Integrating Learning Technologies with Natural Language Processing, Collaborative Research: EAGER: Developing and Optimizing Reflection-Informed STEM Learning and Instruction by Integrating Learning Technologies with Natural Language Processing, G4112919541
OpenAlex ID: W4412889233