Work

A comparative pilot study of the accuracy consistency and marking scheme responsiveness of AI chatbots and human evaluators in professional accounting assessment

Year: 2026

Type: article

Abstract: This pilot study investigates whether artificial intelligence (AI) can effectively assess professional accounting examination scripts comparably to human expert evaluation. The research examines Claude 3.5, GPT-4o, Perplexity, and DeepSeek-V2 across nine subject areas of Ghanaian professional accounting examinations administered by the Institute of Chartered Accountants, Ghana (ICAG). A quantitative experimental design analysed 27 scripts across foundation, application, and professional levels, with each script assessed three times per condition per model, yielding 216 AI-based assessments for comparison with human examiner marks. Statistical analyses included mean absolute deviation (MAD), paired t-tests, chi-square and F-tests, and a novel Marking Scheme Responsiveness Index (MSRI) introduced to quantify AI responsiveness to structured criteria. Claude 3.5 demonstrated the closest alignment with the reference examiner’s marks under guided conditions, achieving a MAD of 4.1 points and a 46.1% improvement over its unguided condition (Cohen’s d = 1.12, 95% CI [0.52, 1.72]); however, as a single uniform prompt was applied across all models, this comparative advantage should be interpreted with caution given the well-documented sensitivity of large language model performance to prompt design. Conversely, GPT-4o showed deteriorating alignment under guided conditions, a counterintuitive finding that warrants further investigation. It is important to note that all AI scores are benchmarked against the marks of a single reference examiner, and the absence of human inter-examiner reliability data limits the interpretability of AI-human deviations. Results suggest assessment structure and model selection may be more influential than subject complexity in determining AI-human score alignment. Four contributions are made: the first empirical benchmark of four state-of-the-art large language models (LLMs) in professional accounting credentialing; operationalisation of the MSRI as a reusable metric; hypotheses challenging prevailing assumptions about AI in professional credentialing assessment; and a validated protocol for future large-scale investigation. All findings are hypothesis-generating, with confidence intervals reported throughout to guide replication planning.

Source: Discover Education

DOI: 10.1007/s44217-026-01837-8