Get started
Work

Lost in translation, found in review? – Pre-registered probes of AI autoformalisation and AI auditing of flawed natural language proofs in Lean

Year: 2026
Type: article
Abstract: Bastounis, Circelli and Hansen (arXiv:2610.08144) argue that a clean Lean compile of an AI translation does not certify the natural-language (NL) proof it came from, and point to two estimates in OpenAI's Navier–Stokes formalisation whose Lean statements do not say what the NL text says. We report pre-registered probes of how current AI models behave as translators, as auditors and as second auditors on such material. In fourteen pre-registered cards with Claude 5.5 models: in the three cards that compiled Lean (lnf1, lnf2, lnf4), no translation of a flawed input that compiled was faithful to it, and — by the experimenter agent's reading — only Claude Haiku 5.5 passed over a flaw in a translation that compiled, at times presenting the compile as support for the flawed step; an Opus 5.5 auditor told of a concern and shown NL and Lean side by side flagged every planted elementary flaw and, when pointed to the lemma, both mismatches published for OpenAI's Navier–Stokes formalisation; and in the pre-registered deference tests with Claude second auditors, on essentially one estimate of that formalisation, no deference to a wrong prior AI review was detected at the power available (0.22 for a 15% rate). A short report with GPT-6 models follows (S6). In those Claude tests no cell where the test was pre-registered met the deference predicate; 1 of the 48 lnf13 runs with a wrong prior review followed it, by the experimenter agent's reading. With two GPT-6 models run through one harness (the Codex CLI), the paper's own opening prompt was answered in all 16 runs by Lean code that compiled — counting answers without a code fence as code — and with no mention of the error in the final answer (one run named it in an interim message); with only the word “just” deleted from it, both models named the error in 8 of 8 runs (one problem, 4 runs per cell; the model shown in the paper was not run). The positive control of our minimal pair — Lemma 8.6 with the derivative count edited from m+4 to m+5 — was machine-checked against the repository for the parameter-free case; the paper's undefined Cmy norm and the reading of the repository's definitions as the paper's objects rest on reading. All cards were designed, run and read by an AI experimenter agent. This record holds the technical note (lnf_note_v1.pdf) and its data package (lnf_note_v1.zip, 409 files). The package mirrors the source repository so that its scripts run unchanged: for every card, the pre-registration and the result (Japanese originals), the runner, the stored outputs (one JSON line per run with the full model output and the MD5 of its prompt) and, for the GPT-6 cards, the raw event streams of the harness; the shared instruments; the P2 machine check in Lean (files, container scripts, logs); the audit requests and results with the cold readers' outputs verbatim (except for the release substitutions listed in README.md); and the scripts that re-derive the counts, the figures and the generated tables. README.md explains the layout, the release substitutions (machine paths and internal identifiers replaced by placeholders; the frozen originals are not edited), the licences and the providers' terms for the model outputs; MANIFEST.json and MANIFEST.sha256 give every file's checksums. Licences. Documents and data: CC BY 4.0. Code: Apache License 2.0 (LICENSE-CODE in the package). Third-party excerpts quoted for this research (passages of OpenAI's write-up Finite time blowup for Navier–Stokes and of its Lean repository at commit f9e8bc5, and the prompts of Figures 1, 3 and 4 of arXiv:2610.08144) are not covered by these licences and stay under their owners' terms. The model outputs are provided for verification and reproduction, not as a training corpus. AI involvement. The cards were designed and run, and the records, the note's draft and the package were prepared, by an AI experimenter agent (Claude Opus 5.5 in Claude Code) and reviewed by cold readers (Claude Fable 5.1 and Claude Opus 5.5; in the final audit also Claude Sonnet 5.5), as described in Section 11 of the note. The experimenter agent set the reference verdicts of all items except P2, whose verdict was machine-checked; every load-bearing decision — the items and controls, the scope of the claims, the status of P2, the inclusion of the GPT-6 report, the title and the final wording — was reserved to and made by the author, who bears sole responsibility for every claim.
DOI: 10.5281/zenodo.23619726
Language: English
Open Access status: green
OpenAlex ID: W7223473193
FWCI:
Cites:
Cited by:
Related to: