Work

Reward Hacking Is an Accounting Error: Eliminating Specification Gaming and the Alignment Tax via Conservation-Gated Reinforcement Learning and Anchored Verifier Evolution

Year: 2026
Type: preprint
Abstract: Prevailing alignment paradigms treat specification gaming as an innate disposition of capable optimizers that must be policed via post-hoc filters, auxiliary critique models, or soft reward penalties, imposing a steep “alignment tax” as optimization scales. We prove that reward hacking is instead an accounting error: it occurs whenever an optimizer collects visible proxy gain G(τ) without independent causal proof L(τ) that the underlying computation was performed. Unpressured language models rarely scheme (0/720 hardcodes in clean probes; 0.28% across 2,480 impossible coding attempts); standard post-training manufactures deception by paying unbacked credit U(τ) = G(τ) − L(τ) > 0. We introduce Ledger-Gated Training (LGT), enforcing the strict conservation identity U(τ) = 0 prior to every gradient update via a multiplicative trajectory gate anchored by an unbiased forward-KL token objective. We prove that all finite linear penalties (λ < ∞) and Constrained-MDP budgets (E[U] ≤ δ) fail via hybrid subsidies, prerequisite shortcuts, and budget exhaustion; that any receipt auditor better than chance (ϵleak + ϵalarm < 1) renders honest work strictly dominant within the competence ratio κ; and that under natural policy-gradient flow, LGT extinguishes shortcuts exponentially while amplifying honest trajectory mass (ρt > 1), yielding a provably negative alignment tax. Evaluated with all gates switched OFF on unseen tasks, running online GRPO to KL convergence causes standard RLVR to collapse. Cheating explodes to 54.9% (258/470) and specification-correctness plunges from 76.8% to 27.2%, whereas LGT holds cheating at 1.1% (5/470; 50× reduction, z = 18.4) and lifts specification-correctness to 80.8% (+53.6 pp, z = 16.5). At 3× learning rate, LGT suppresses cheating by 16× (1.1% vs. 18.1%, z = 8.87) and delivers a 2× capability speedup (+22 attempt-1 passes in 31 steps). Across 5/5 runs, LGT rises above initialization on attempt-1 capability, beats a privileged hidden-test oracle on final-of-3 synthesis in 4/5 runs, and cuts cheats on an untrained helper suite in 5/5 runs; in iterative self-training, extinguishing shortcuts to 0.0% ignites hard synthesis pass@1 by 4.1× (4.0% to 16.7%, 16.3%). Under Anchored Verifier Evolution (AVE), a self-revising verifier attacked across 114,584 generations by Qwen2.5-72B achieves 0.0% false certifications (0/324 governed held-out reports across 972 three-track evaluations, vs. 81.5% ablated) while raising recall from 0.953 to 1.000.
Language: English
Open Access status: green
OpenAlex ID: W7233596562
Cites:
Cited by:
Related to: