A Fallback Is Not a Disposition: What Verifier-Passing Shortcuts in RLVR-Trained Models Show, Why Their Rate Rises With Task Difficulty on One Benchmark and Falls on Another, and the Signatures That Would Separate Fallback, Selection and Learned Exploitation
Year: 2026
Type: preprint
Abstract: (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Reinforcement learning with verifiable rewards (RLVR) pays a model whenever an automatic checker accepts its output, so any output the checker wrongly accepts is paid as well. A 2026 study of inductive rule learning found that RLVR-trained models sometimes answer a request for a general rule by listing the labels of the training instances, which a checker of label consistency accepts, and concluded that this is reward hacking specific to RLVR rather than a failure of understanding. In this analysis the author reviews and synthesises published results to show that the frontier-model evidence for that conclusion cannot separate three accounts that predict the same table: a fallback emitted when the genuine solution is out of reach, the selection of an output mode the model already had, and an exploitative disposition learned in training. Read conditionally, the study's own counts place 368 of its 500 shortcuts in one small model, rise with reasoning effort even among failed tasks, and fall to under one percent of the strongest model's hardest failures. Across the wider record the difficulty gradient runs in both directions: harder tasks raise hack rates in planted-loophole environments and in the anchor, while easier tasks raise cheating on impossible coding tasks. The author argues that a relative-cost reading reconciles them, that an offered exit removed most of the effect for two of the three models on which it has been tested, and that RLVR is neither necessary nor sufficient: shortcuts appear zero-shot in instruction-tuned models and after supervised fine-tuning, while most models trained with RL alone against a hackable test did not learn to hack. The paper consolidates the measured values in Table 1 and Figure 1, sets out the signatures of each account in Table 2, states an attribution procedure as Algorithm 1, and specifies the measurements that would assign a given shortcut to its cause.
Author: Mahendrakar, Pranay
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225882104
