Year: 2026
Type: conference-paper
Abstract: Group Relative Policy Optimization (GRPO) and its variants are effective for reinforcement learning with verifiable outcome rewards, but incorporating intermediate credit remains difficult.We show that, under practical sampling budgets, token-level Monte Carlo estimates at early reasoning positions have high variance; group-relative normalization turns this noise into frequent advantage-sign errors and wrongdirection gradients.Later estimates instead approach the terminal reward and add little information.We propose SCRoll, a rollout-side module that adds randomized single-boundary segment rewards without changing the downstream optimizer.An upstream stream generates an early-stopped segment s, and a downstream stream samples G continuations conditioned on [s, q], placing the segment before the original problem.The shared downstream group supplies both terminal outcomes for any group-relative objective and the upstream segment's Monte Carlo reward, avoiding a separate rollout group; a single-stream objective optimizes the upstream segment.Across Qwen2.5-Math-7B and Llama-3.2-3B-Instruct,SCRoll improves most pairings with GRPO, GSPO, DAPO, and CISPO, with the clearest gains for CISPO.A decoupled two-process baseline reaches comparable performance only with substantially more generated tokens, while SCRoll performs better under a matched token budget.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224251169