Work

Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay

Year: 2026

Type: conference-paper

Abstract: Reinforcement learning (RL) is effective for improving code generation but suffers from data scarcity.While experience replay mitigates this, existing approaches rely on static, in-epoch metrics that overlook training dynamics, often introducing low-utility or outdated data.Analyzing RL dynamics via dataset cartography, we observe that "ambiguous" samples, which are vital for model generalization, rapidly migrate to "easy-to-learn" regions, diminishing their training value.To address this, we propose Adaptive Ambiguity Replay (A2R) for RL, a plug-and-play module that prioritizes cross-epoch ambiguous samples.To neutralize the noise from stale experiences, A2R incorporates an adaptive importance mechanism based on policy divergence to weigh replayed rollouts.Extensive experiments on nine LLMs (3B-14B) demonstrate that A2R outperforms state-of-the-art baselines on real-world code editing tasks across both unseen and learned domains.Our results highlight cross-epoch ambiguity as a key factor for effective replay in RL.Code

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.886