Work

Toward a Verification Singularity : Quantifying the Verification Gap and Engineering Minimum Viable Guardrails for AI-Generated Code

Year: 2026
Type: dissertation
Abstract: Background. The widespread adoption of AI-assisted code generation tools has created a structural asymmetry in software engineering: code can be synthesised in minutes, but human reviewers still require hours or days to audit it. This thesis introduces the Verification Chasm, the growing gap between AI generation velocity and human verification capacity, and proposes the V-Ratio (V = Tverify/Tgenerate) as a formal indicator of that gap. Because AI-generated code is not merely faster-produced human code — it carries a documented tendency toward semantically plausible yet logically incorrect outputs — the chasm is not a productivity inconvenience but a systemic reliability threat. Objectives. Three sequential research questions are pursued. RQ1 empirically quantifies the Verification Chasm using the V-Ratio, computed from 372 AI-assisted pull requests across 11 production repositories over 37 months. RQ2 characterises the structural failure modes of deterministic unit testing against three categories of AI-generated hallucinations through a controlled adversarial mutation experimenton 717 mutations from 239 baseline functions. RQ3 designs, iterates, and evaluates Z-SAAP, the Zero-Sycophancy Adversarial Audit Pipeline, a fully deterministic four-stage guardrail engineered to intercept the hallucinations that RQ2 proves testing alone cannot catch. Methods. RQ1 employs a survey-based research design using observational, repository-mined data, with Spearman rank correlation, OLS regression, K-Means clustering(k=3), and Kruskal-Wallis analysis across 37 months of GitHub data. RQ2 uses acontrolled adversarial mutation experiment with a pre-specified Scope of Impact taxonomy and Fisher’s Exact Test. RQ3 is an implementation study, structured by the Design Science Research (DSR) framework, evaluating six pipeline versions across a 956-case corpus on two primary metrics: Safety Score and False Positive Rate. Results. RQ1 finds a median V-Ratio of 166.4× (n=372), with 45.7 % of pull requests in the Critical or Extreme tier. Claude Cognitive Score is the strongest single predictor (r=+0.243, p<0.0001); each one-point increase is associated with an estimated 17.0 additional hours of verification time. RQ2 establishes an overall Detection Failure Rate of 34.5 %, identical across five independent runs (std = 0.00 %), which indicates that the bypass is structural rather than a stochastic artefact. Fisher’s Exact Test confirms Systemic Logic Faults are 1.46× more likely to evade detection than Localised faults (p=0.030). RQ3 develops and evaluates six successive guardrail architectures, charting the detection–precision trade-off as the design moves from LLM-based debate to a fully deterministic approach (Z-SAAP v6) that records a Safety Score of 49.91 % and a False Positive Rate of 22.03 % at a mean latency of 1.26 seconds, including 100 % detection of Stochastic Output faults but a bounded detection rate on the hardest surviving faults; the sequence is presented as a line of design reasoning rather than as a finished guardrail. Conclusions. The findings collectively establish that the Verification Chasm is measurable, that standard testing is structurally insufficient to close it, and that a deterministic external guardrail can intercept a meaningful share of the semantic faults that escape both LLM alignment and unit testing. The V-Ratio, the Scope of Impact taxonomy, and Z-SAAP v6 constitute transferable contributions toward a verification infrastructure commensurate with modern AI generation velocity.
Language: English
Open Access status: green
OpenAlex ID: W7202334293
Cites:
Cited by:
Related to: 10