Year: 2026
Type: conference-paper
Abstract: Recent preference optimization algorithms such as Direct Preference Optimization (DPO) have become prevalent for aligning large language models (LLMs) with human preferences.FocalPO improves upon DPO by introducing a modulating factor that down-weighs misranked preference pairs.However, using a fixed modulating factor throughout training is suboptimal, as the model's learning capacity evolves during training.We introduce DynamicFocalPO, which employs a dynamic focusing strategy that adapts over the course of training.Inspired by curriculum learning, our method initially focuses on correctly ranked samples to establish a solid foundation, then gradually incorporates harder samples as training progresses.Experiments demonstrate that DynamicFocalPO surpasses both DPO and FocalPO on benchmarks including Alpaca Eval 2.0 and Arena-Hard using Mistral-Base-7B and Llama-3-Instruct-8B.We further provide theoretical analysis showing that the dynamic schedule enables adaptive entropy regularization and selective gradient suppression.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166886684