Get started
Work

SmolLab_SEU at BEA 2025 Shared Task: A Transformer-Based Framework for Multi-Track Pedagogical Evaluation of AI-Powered Tutors

Year: 2025
Type: conference-paper
Abstract: The rapid adoption of AI in educational technology is changing learning settings, making the thorough evaluation of AI tutor pedagogical performance is quite important for promoting student success.This paper describes our solution for the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered tutors, which assesses tutor replies over several pedagogical dimensions.We developed transformer-based approaches for five diverse tracks: mistake identification, mistake location, providing guidance, actionability, and tutor identity prediction using the MRBench dataset of mathematical dialogues.We evaluated several pre-trained models including DeBERTa-V3, RoBERTa-Large, SciBERT, and EduBERT.Our approach addressed class imbalance problems by incorporating strategic fine-tuning with weighted loss functions.The findings show that, for all tracks, DeBERTa architectures have higher performances than the others, and our models have obtained in the competitive positions, including 9 th of Tutor Identity (Exact F1 of 0.8621), 16 th of Actionability (Exact F1 of 0.6284), 19 th of Providing Guidance (Exact F1 of 0.4933), 20 th of Mistake Identification (Exact F1 of 0.6617) and 22 nd of Mistake Location (Exact F1 of 0.4935).The difference in performance over tracks highlights the difficulty of automatic pedagogical evaluation, especially for tasks whose solutions require a deep understanding of educational contexts.This work contributes to ongoing efforts to develop robust automated tools for assessing.
DOI: 10.18653/v1/2025.bea-1.88
Language: English
Open Access status: hybrid
OpenAlex ID: W4412889182
FWCI: 1.059
Cites:
Cited by: 1
Related to: 10