PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
Year: 2025
Type: preprint
Abstract: Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities,particularly in environments with complex reward landscapes.
Source: arXiv (Cornell University)
Authors: Driss, Brahim, Davey, Alex, Akrour, Riad
Institutions: Centre National de la Recherche Scientifique, Université de Lille, Centre de Recherche en Informatique, Signal et Automatique de Lille, SCOOL: Prise de décision séquentielle dans l'incertain, Centrale Lille Institut, Université Paris-Saclay, Laboratoire Interdisciplinaire des Sciences du Numérique
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4414889805
Cites:
Cited by:
Related to: