The Audit Regime Gap: Regime-Conditional Bias Evaluation in Large Language Models
Year: 2026
Type: preprint
Abstract: This pre-registration documents the experimental protocol for "The Audit Regime Gap," a study investigating whether anti-Muslim bias measurement in large language models depends on the inference regime used (completion, reasoning, or agentic decision-making). Purpose: We formalize bias as a functional of a model, item distribution, and inference regime, deriving three estimable quantities: the audit validity gap (G), the audit transfer coefficient (τ), and the mitigation misranking index (M). We instantiate this framework in MIRAGE, a benchmark of 1,200 minimal-edit matched-pair prompts in five identity variants (Muslim, Christian, Jewish, Hindu, secular), evaluated under four inference modes (C₁ direct completion, C₂ₐ elicited chain-of-thought, C₂_b native reasoning, C₃ agentic decision) crossed with four retrieval-recency conditions (empty, neutral, historical, recent), with a 400-prompt parallel Arabic subset and a placebo-swap null calibration. Primary Hypotheses (pre-registered): H1 (Reasoning amplification): B_C₂ₐ > B_C₁ and B_C₂_b > B_C₁ for all models; the amplification index exceeds 1. H2 (Native > elicited): B_C₂_b ≥ B_C₂ₐ: models trained to reason natively produce longer intermediate output upstream of the output-stage suppressor. H3 (Weak transfer to decisions): τ_C₁→C₃ is substantially below τ_C₁→C₂ₐ. H4 (Misranking with consequence): M_C₁→C₃ > 0 and the induced selection regret is strictly positive. H5 (Suppression depth predicts amplification): In reasoning-capable open models with probe access, later suppression onset correlates positively with the amplification index. H6 (Recall-bound agreement): Empirical top-q audit recall agrees with the theoretical bound within ±5 pp at q ∈ {0.05, 0.10, 0.20}. Expected Outcomes: We predict that reasoning amplifies rather than suppresses Muslim–violence association, that agentic decision asymmetry persists under prompt-based defences, that bias tracks retrieval recency, and that a completion-only audit inspecting the worst decile recovers only a fraction of the worst decile under agentic decision. Models Evaluated: GPT-4o, Claude Opus 4, Mistral Large, Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct, DeepSeek-R1. Statistical Protocol: Paired bootstrap with B=10,000 prompt-level resamples stratified by template, with bias-corrected and accelerated adjustment where bootstrap skewness exceeds 0.5. Holm-Bonferroni correction applied within pre-declared families. Time-coupling coefficient tested via permutation test shuffling pool labels within prompt.
Author: Noor Islam S. Mohammad
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7204265183