Work

Preliminary evaluation of an AI-powered chatbot for academic integrity support in higher education

Year: 2026

Type: article

Abstract: This study reports a preliminary internal evaluation of the A-Integrity Buddy chatbot, an AI-powered tool designed to provide policy-aligned academic-integrity guidance within one Australian university. Using a scenario-based quantitative design, 84 author-developed scenarios grouped into author-assigned Easy (n = 30), Medium (n = 27), and Difficult (n = 27) categories were submitted to the chatbot. The first generated response to each scenario was retained and rated separately, without consultation, by three institution-based experts on a five-point scale for policy alignment and response adequacy. The same expert panel had contributed to scenario development; accordingly, the study represents an internal formative benchmark rather than independent validation. Across 84 scenarios, the overall mean scenario rating was 4.56 out of 5 (95% bootstrap interval [4.45, 4.65]). Mean scenario ratings were 4.71 (95% bootstrap interval [4.60, 4.81]) for Easy, 4.43 (95% bootstrap interval [4.16, 4.68]) for Medium, and 4.52 (95% bootstrap interval [4.40, 4.63]) for Difficult scenarios. Overall ordinally weighted Gwet’s AC2 was 0.911 (95% CI [0.886, 0.935]), although exact three-rater agreement occurred for 39.3% of scenarios. The high weighted coefficient therefore reflected agreement or near-agreement, predominantly within the upper rating categories. These findings indicate generally favourable internal expert appraisal of the fixed chatbot outputs within the evaluated institutional policy context. Interpretation is constrained by the small non-independent rater panel, the single-institution setting, and the use of one generated response per scenario. Further independent and end-user evaluation is required before claims can be made about reliability, educational impact, or generalisability. Not applicable.

Source: Discover Education

DOI: 10.1007/s44217-026-02381-1

Authors: Thinh Huynh, Tommy Huynh, Amanda Shaker, Haider Al Abadi

Institutions: Deakin University, La Trobe University

Language: English

Topic: Academic integrity and plagiarism

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDG: Quality education

Open Access status: gold

Estimated APC paid: $1,390