When Is an RL Problem a Diffusion Problem? A Ten-Domain Empirical Map
Year: 2026
Type: preprint
Abstract: Scramble inversion -- train a denoiser to predict the action that undoes the last step of a random walk from the goal, then act by reverse-process rollout -- beats deep approximate value iteration (DAVI) on the Rubik's Cube at a fraction of the compute. Does "every RL problem can be a diffusion problem"? We answer with a ten-domain assumption ladder, each domain deleting one property of the cube: group structure (24-puzzle), fixed goal and environment (procedural mazes), invertible actions (Sokoban), discreteness (pendulum), determinism (2048), bounded horizon (Tower of Hanoi, optimal solutions of 1023 moves), full observability (POMDP mazes), ordered solutions (Lights Out), monotone progress (Mountain Car), and conservation (Peg Solitaire). Every domain is run under a matched architecture, matched-compute protocol against a value baseline on one consumer GPU, with exact oracles wherever the state space permits and mechanical replay verification of every claimed solution. The resulting map is sharp and mechanistic. Scramble inversion dominates at scale: 100.0% vs. 0.6% on the 24-puzzle (7.7e24 states) at matched iterations. It survives conditioning, partial observability, commutativity, and non-monotone dynamics. It has two repairable failure modes: with irreversible actions its reverse-only data is deadlock-blind (Sokoban: 65.2% vs. DAVI's 82.7%), which a zero-training hybrid -- denoiser proposes, value function vetoes -- not only repairs but turns into the best solver (93.0%); and with continuous actions, naive epsilon-regression collapses to a useless posterior mean (0.0% swing-up) while the same data with a distributional head reaches 100.0%. Extreme horizon (Hanoi) exposes a third, diffusion-native failure shared by both methods: non-backtracking noising walks never mix across the 1023-move diameter (1200-step walks stay within distance 125), and each method is near-perfect inside the covered shells and zero beyond -- the schedule, not the learner, is the ceiling. Finally there is one hard boundary: stochastic dynamics (2048), where the deterministic reverse process describes a game that is never played and the denoiser performs at random-play level -- and a spawn-aware noising walk that matches the state marginal does not help, because the undo-labels themselves answer the wrong question. We conclude with a practitioner's checklist and release all ten environments, trainers, and logs.
Author: Khani, Aamer
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203883583
Cites:
Cited by:
Related to: