Do Verification Tools Improve on an AI Reader? A With-and-Without Evaluation of Fifteen Open-Source Tools for AI-Assisted Research
Year: 2026
Type: preprint
Abstract: Background. AI agents are often given checking tools to improve their work, but there is little evidence that these tools do better than an AI reading the same material alone. Objective. To test fifteen open-source verification tools for AI-assisted research and document work, with and without each tool, and to show how the choice of comparison can change the conclusion. Methods. Four experiments were run on 14 and 15 October 2026 using Codex and Claude Code. In Experiment A (a pilot), each tool was scored on 20 cases derived from its own rules and compared with a null comparator that accepted every case (300 cases). In Experiment B, each tool was compared with an unaided AI reader on 450 independently built artifacts (4,568 items; 30 artifacts per tool), using Welch t-tests with Holm correction, exact McNemar tests and random-effects pooling. Experiments C and D tested a writing skill (22 tasks, two models) and a specification checker (20 briefs). Results. In Experiment A, the tools classified 98.0% of cases correctly, against 50.0% for the null comparator (p < .001). In Experiment B, mean accuracy per artifact was 0.62 with the tools and 0.97 for the reader (pooled difference −0.35, 95% CI −0.43 to −0.26; I² = 98%). The reader was better for all 15 tools after correction (Holm p ≤ .021). Within each tool’s documented scope, the tool was worse for 11 tools, level for 3 and better for 1. Over all items, the tools flagged 48% of defects and left 90% of clean items alone, against 94% and 99% for the reader. The tools gave no verdict on 21.4% of items, whereas the reader gave no verdict on 3 of 4,568. Two tools (specguard and releaseproof) were correct on every item they decided. Adding a tool to the reader improved accuracy for 2 tools, made no reliable difference for 6 and lowered accuracy for 7. The writing skill reduced meta-writing from 8 of 20 responses to 0 of 20 (Fisher exact p = .003), and no model obeyed a planted instruction with or without it. The specification checker flagged all 11 failing requirements in the drafting test, with no false alarm and no miss. Conclusions. The pilot and the head-to-head experiment gave opposite verdicts on the same tools, so the choice of comparator and test cases can decide the conclusion. The tools were most useful where a task needs calculation or exhaustive cross-checking, and least useful where false alarms outweighed extra catches. Independent replication is invited. Version 2 adds the full evaluation data and code as one archive (Trust-Tools-Evaluation-Data-and-Code.zip): corpora, truth files, verdict files, raw tool outputs and analysis scripts for Experiments A to D, with a README and a SHA-256 manifest. The paper text is updated to say so.
Author: Oluwawalemi, Oluwaseun John
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7222189691
Cites:
Cited by:
Related to: