Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment
Year: 2025
Type: conference-paper
Abstract: Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach.However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses.This results in conflicting optimization directions, hindering the optimization on the Pareto Front.To address this, we propose to construct Paretooptimal responses to resolve preference conflicts.To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for selfsupervised preference alignment.Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines 1 .
Authors: Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi, Zhuo Liu, Fuli Feng, Tat-Seng Chua
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888334