AI as a Junior Assessor: Can Large Language Models Score IFRS S2 Climate Disclosures? Evidence from Nigerian Upstream Oil and Gas
Year: 2026
Type: preprint
Abstract: Nigeria's Financial Reporting Council requires public interest entities to report under IFRS S1 and IFRS S2 from 1 January 2028, and ISSA 5000 applies to sustainability assurance engagements for periods beginning on or after 15 December 2026. Audit firms with few climate-trained staff are considering large language models (LLMs) for first-pass disclosure review. This paper asks whether LLMs can score climate disclosures the way a trained human assessor does. Four models scored the FY2025 reports of Seplat Energy, Oando and Aradel Holdings on 15 IFRS S2, SASB and NUPRC indicators and were compared with a published human benchmark. The models received the indicator names and a general 0 to 3 scale, without the coder's indicator-specific criteria, which mirrors how a small practice would first use an off-the-shelf model. DeepSeek V4.1 Flash and Gemini 2.5 Flash read the full reports, up to 443,000 billed prompt tokens, three times each; GPT-4o and Claude Sonnet 4.5 read one evidence pack per firm. In single runs, Claude Sonnet 4.5 and DeepSeek V4.1 Flash reached moderate agreement with the human coder within firms (quadratic weighted kappa 0.53 and 0.51) and came within one level of the human score on more than 90% of items. Across all three firms, a naive rule that gives each firm its typical human score on every indicator matched or beat every single run on pooled agreement and error size, so pooled agreement mostly reflects telling strong reporters from weak ones. On the two firms where every model received complete input, Claude and DeepSeek beat that rule. The median of three DeepSeek runs beat that rule on every measure (within-firm kappa 0.64, mean absolute error 0.29). Gemini 2.5 Flash gave 82% of items the same middle score; it repeated itself almost perfectly and agreed least with the human. The largest errors came from the pipeline: an evidence pack cut short by a character limit, and one run that lost track of a 368,000-token report. LLMs can locate evidence and draft scores for Nigerian readiness work, provided firms validate them on within-firm agreement, control what reaches the model and review every score. Human benchmark: Davies (2026a), https://doi.org/10.5281/zenodo.23984011Replication package (prompts, rubric, raw logs, code): https://doi.org/10.5281/zenodo.24046408
Author: Davies, Joseph
Language: English
Subfield: Strategy and Management
Domain: Social Sciences
SDG: Climate action
Open Access status: green
OpenAlex ID: W7225596470