Year: 2026
Type: conference-paper
Abstract: Policy-gradient reinforcement learning (RL) is widely used to improve language model reasoning, but existing methods are not compatible with diffusion language models.The primary reason for this is the difficulty of likelihood estimation with such models.We propose EMBR , a scalable off-policy framework that reformulates KL-regularized RL as an energy-based distribution matching problem.By aligning policy updates with reward signals through energy matching, EMBR avoids the overhead of on-policy learning and the variance of importance weighting.We further derive a principled upper bound for the energy matching objective which can be used to fine-tune dLLMs.Experiments on multiple benchmarks in both online and offline setting show that EMBR matches or surpasses the performance of GRPO and related baselines in the online case, and of DPO in the offline case.Our approach provides a practical alternative for post-training of diffusion LMs.
Author: Shiv Shankar
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140081529