Year: 2025
Type: conference-paper
Abstract: Large Reasoning Models (LRMs) such as Ope-nAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling testtime compute and generating long Chain-of-Thought (CoT).Distillation post-training on LRMs-generated data is a straightforward yet effective method to enhance the reasoning abilities of smaller models, but faces a critical bottleneck: we found that distilled long CoT data poses learning difficulty for small models and leads to the inheritance of biases (i.e.over-thinking) when using Supervised Fine-tuning (SFT) and Reinforcement Learning (RL) methods.To alleviate this bottleneck, we propose constructing tree-based CoT data from scratch via Monte Carlo Tree Search (MCTS).We then exploit a set of CoTaware approaches, including Thoughts Length Balance, Fine-grained DPO, and Joint Posttraining Objective, to enhance SFT and RL on the construted data.We conduct evaluation on various benchmarks such as math (GSM8K, MATH, AIME).instruction-following (Multi-IF) and planning (Blocksworld), results demonstrate our approaches substantially improve the reasoning performance of distilled models compared to standard distilled models via reducing the hallucinations in long-time thinking.The project homepage is https://github.com/ AIDC-AI/Marco-o1.
Authors: Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Huaiyu.wh Huaiyu.wh, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, Weihua Luo, Kaifu Zhang
Institutions: Alibaba Group (China), Tsinghua University, Monash University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4412944736