Year: 2026
Type: article
Abstract: The use of artificial intelligence (AI) to support assessment and feedback in higher education has attracted growing interest, but concerns remain about the consistency and appropriateness of AI-generated grades in summative contexts. This study examined whether a combined calibration strategy involving a more explicit rubric and marking prompt could improve agreement between AI-assisted grading and educator judgement in a laboratory-based STEM assessment. We compared educator-assigned marks for 40 graduate Electrochemical Energy Storage Systems laboratory reports with marks generated by GPT-5 under pre- and post-calibration configurations, and examined agreement using complementary statistical and visual analyses. Following calibration, agreement with educator marks improved across measures of score correspondence and absolute agreement: the intraclass correlation coefficient increased from 0.818 to 0.928, the mean AI-educator difference decreased from +1.625 to +1.041 marks, and no post-calibration report differed from the educator mark by more than 10 marks. These findings suggest that the way assessment criteria, evidence requirements, and marking rules are specified can materially influence AI-human grading agreement. However, the revised rubric and prompt were introduced together, calibration drew on discrepancies observed in the same 40 reports used for post-calibration evaluation, and one educator's marks served as the reference. The results therefore support improved in-sample agreement under a human-supervised calibration strategy, rather than establishing grading validity or interchangeability for autonomous summative decisions.
Source: Frontiers in Education
Authors: Deepshikha Deepshikha(Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing), Navpreet Singh(Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – review & editing)
Institution: Queen Mary University of London
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: gold
Estimated APC paid: $2,353
OpenAlex ID: W7236019175
