When the Rules Run Out: Can language models find what aircraft certification rules miss? Evidence from 32 years of FAA special conditions
Year: 2026
Type: preprint
Abstract: Every genuinely new aircraft feature reaches a point where the airworthiness rules have no answer. The FAA then writes special conditions, and spotting these gaps early is one of the hardest and most consequential steps in certification. Language models are being proposed as assistants for this work, but special conditions are public and 86% of them repeat earlier ones, so a model can look capable simply by recalling precedent. We built CertBasis-Bench, a benchmark whose answers were written by the FAA itself, to separate the two. We tested three open-weight model families (five configurations) and five closed models from OpenAI and Anthropic on the same 314 items, plus one decision model under its own protocol. At first, the models appeared to beat a precedent-based baseline on first-of-kind features. A closer look showed that this advantage comes from items published before 2010, mostly a 2006 cluster of Airbus A380 items, where no precedent existed to compare against. From 2010 onwards, every model scored below what copying later special conditions for the same feature would give. The best closed model, GPT-6.1 Sol, has the highest overall score, but its lead over the precedent template is not statistically clear, and it does not lead on new features. We find no evidence that current open-weight or closed models identify new certification gaps beyond precedent. They are useful elsewhere: given the official text, they answer "is this rule still in force?" correctly 77–89% of the time, against 30–60% without it. Funding: This work received no specific grant from any funding agency. Compute (cloud model API credits) was provided in kind by Industrial AI Lab. Competing interests: G. Gonçalves is Vice President of Industrial AI Lab (non-profit) and owns SmartHow, a company that provides AI solutions for industry. SmartHow could in future build or sell AI tools for certification or compliance work of the kind this benchmark evaluates. No SmartHow product is evaluated. AI use: Claude Code (Anthropic; Claude Opus 5.5) wrote the pipeline, scoring code and figures, drafted the synthetic T1 items and the paper; the novelty threshold was calibrated against Claude as coder; the SC-25 families and labels were produced by an AI agent. AI-only; no human verification. AI is not an author.
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225200248