Year: 2026
Type: conference-paper
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs' short-context reasoning but falters in long-context scenarios requiring precise grounding and multi-hop reasoning.We identify the "almost-there" phenomenon-trajectories that are largely correct but fail at the final step-in long-context reasoning RL and attribute this failure to two factors: (1) the lack of high reasoning density in long-context QA data, and (2) indiscriminate penalization of partially correct trajectories during long-context RL.To overcome this bottleneck, we propose DEEPREA-SONQA, a KG-driven synthesis framework that controllably constructs high-difficulty, multihop long-context QA pairs with inherent reasoning chains.Building on this, we introduce Long-context Process Advantage Shaping (LONGPAS), a simple yet effective method that performs fine-grained credit assignment by measuring reasoning steps along Validity and Relevance dimensions, which captures critical signals from "almost-there" trajectories.Experiments on three long-context reasoning benchmarks show that our approach substantially outperforms RLVR baselines and matches frontier LLMs while using far fewer parameters.Further analysis confirms the effectiveness of our methods in strengthening long-context reasoning while maintaining stable RL training.Codes are available
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166870774