Work

Quasi-experimental evaluation of a human-centric AI tutoring and human-in-the-loop assessment system in advanced STEM education

Year: 2026
Type: article
Study design: Clinical Trial
Abstract: Artificial intelligence is reshaping higher education, yet most deployments of large-language-model tutoring and automated grading target introductory courses and pay limited attention to the transparency, trust, and equity concerns that become acute in advanced, safety-critical technical domains. This paper presents a human-centric AI-assisted instructional system and evaluates it in a postgraduate hardware-security course on electromagnetic protection and side-channel analysis. The system couples three human-centric components: (i) a retrieval-augmented tutoring assistant whose answers are grounded in citable course materials to curb hallucination and preserve traceability; (ii) a human-in-the-loop assessment pipeline in which a language model drafts rubric-based feedback that instructors verify item by item; and (iii) an adaptive, tiered project-based learning path that offers differentiated entry points. We evaluated the system in a quasi-experiment with 47 postgraduates (24 experimental, 23 control) taught concurrently by the same instructors in parallel class sections. The experimental group’s competency score rose from 62.4 to 84.7, against 61.8 to 70.2 for the control group, a difference in gains that was statistically significant and associated with a large effect size ( \(t(41.2)=4.55\) , \(p<0.001\) , Cohen’s \(d=1.32\) ), while its score dispersion contracted from a standard deviation of 14.5 to 7.2. A baseline-by-group interaction indicated significantly stronger differential improvement among initially lower-scoring students in the experimental section, although this distributional finding is interpreted cautiously given the non-randomised design. Human-in-the-loop grading was associated with a 35.7% reduction in marking time while reaching an AI–instructor agreement of quadratic-weighted Cohen’s \(\kappa =0.95\) ; because this agreement measures consistency rather than correctness, we interpret it accordingly. We examine how human oversight, citation grounding, and adaptivity may have jointly contributed to these outcomes, treating the intervention as an integrated bundle whose individual components remain to be isolated, and we set out the responsible-AI safeguards that such deployments require.
Language: English
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7234403962
FWCI:
Cites: 25
Cited by:
Related to: 10