Work

TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning

Year: 2026

Type: conference-paper

Abstract: Reinforcement learning (RL) has emerged as an effective paradigm for enhancing model reasoning.However, existing RL methods like GRPO typically rely on unstructured self-sampling to fit scalar rewards, often producing inefficient rollouts that fail to capture transferable problem-solving strategies.To address this limitation, we propose TemplateRL, a structured template-guided RL framework that augments policy optimization with explicit template guidance.Our approach first constructs a problem-solving template library via MCTS on a small seed set, then seamlessly integrates this high-level structured guidance into RL training.By guiding rollout generation to align with proven template structures, TemplateRL significantly improves high-quality trajectory hit rates while reducing ineffective exploration.This structure-guided design steers the policy toward validated strategic patterns, stabilizing training dynamics, and enhancing RL sampling efficiency.Notably, the explicit template library is interpretable, editable, and supports online updates-enabling continuous updates during both training and inference.Extensive experiments demonstrate that TemplateRL outperforms GRPO by 99% on AIME and 41% on AMC, with superior stability on weak models and remarkable cross-domain generalization, highlighting its potential for broader tasks.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.198