LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
Year: 2025
Type: conference-paper
Abstract: Recent advancements in large language models (LLMs) have markedly improved their capacity to handle long text inputs; however, current models, including GPT-4o, still exhibit unsatisfactory performance in long-form generation.Generating high-quality long-form content still remains a significant challenge.In this paper, we present LongDPO, a novel approach designed to enhance long-form text generation through step-level supervision.By leveraging Monte Carlo Tree Search (MCTS) to collect stepwise preference pairs and employing a global memory pool to maintain factual accuracy, LongDPO effectively mitigates issues such as inconsistencies that are prevalent in long-context LLMs.Furthermore, we integrate critique-augmented generation to refine the selected preference pairs.Following the collection of stepwise preference pairs, we apply stepwise preference learning for fine-grained optimization.Experimental results demonstrate that our method enhances performance on longform generation benchmarks (e.g.LongBench-Write) while maintaining nearly lossless performance on several general benchmarks. 1 1 Code and models will be publicly available at https: //github.com/pingbowen23/LongDPO.
Authors: Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, Shanghang Zhang
Institutions: Peking University, Tencent (China), Tsinghua University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Award: G1870188661
OpenAlex ID: W4412888641