Year: 2025
Type: conference-paper
Abstract: The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance.However, the recursive thinking capabilities of Large Language Models (LLMs) remain limited, particularly in the absence of expert-curated data for distillation.In this paper, we propose AvR: Alignment via Refinement, a novel method aimed at unlocking the potential of LLMs for recursive reasoning through long-form CoT.AvR introduces a refinement process that integrates criticism and improvement actions, guided by differentiable learning techniques to optimize refinementaware rewards.As a result, the synthesized multi-round data can be organized as a long refinement thought, further enabling test-time scaling.Experimental results show that AvR significantly outperforms conventional preference optimization methods.Notably, with only 3k synthetic samples, our method boosts the performance of the LLaMA-3-8B-Instruct model by over 20% in win rate on AlpacaEval 2.0.Our code is available at Github 1 .
Authors: Haoke Zhang, Xiaobo Liang, Cunxiang Wang, Juntao Li, Min Zhang
Institutions: Soochow University, Soochow University, Tsinghua University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G2027526133, G3114819178, G6441875606
OpenAlex ID: W4412888223