Auditing the Auditor: Extraction Accuracy, Hallucination, and Abstention in Open-Weight LLMs for ESG Reporting A Longitudinal Empirical Benchmark Across 518 Corporate Disclosures (2010–2026)
Year: 2026
Type: preprint
Abstract: Corporate sustainability auditing increasingly relies on Large Language Models (LLMs) to extract verified quantitative metrics from lengthy Environmental, Social, and Governance (ESG) reports. For an LLM to serve as a trustworthy compliance tool, it must pull exact numbers from dense ta- bles while refusing to guess when a disclosure is missing. We evaluate five leading open-weight LLMs (Qwen3-4B, DeepSeek-R1-7B, GPT-OSS-20B, Qwen3.5-27B, and Gemma-4-31B) across 518 corporate sustainability reports published between 2010 and 2026 (279,862 text passages). Using a stratified 120-question audit benchmark tested under five controlled evidence conditions (3,000 total responses), we grade every output using an independent referee model (Phi-4-14B) so that no model grades its own family’s work. Three findings stand out for sustainability practi- tioners: (1) models summarize narrative policies reliably (F1 accuracy ≈ 0.729) but struggle to extract exact numerical indicators such as Scope 1 emissions or water withdrawal (F1 ≈ 0.335); (2) even when handed the exact gold-standard page from a report, the strongest model (GPT- OSS-20B) reaches an accuracy ceiling of 0.731, showing that table-reading errors matter more than search errors; and (3) when given a “near-miss” passage from the wrong year or company, standard prompts cause models to invent or substitute numbers in up to 46.7% of cases. By contrast, step-by-step reasoning in DeepSeek-R1-7B catches the mismatch and abstains 77.5% of the time unprompted, while strict audit instructions raise abstention across dense models to 74.2%–80.0%.
Source: Research Square
Institution: University of Ghana
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225734016