Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
Year: 2024
Type: preprint
Abstract: Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.
Source: arXiv (Cornell University)
Institution: University of Washington
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: IIS:RI Theoretical Foundations of Reinforcement Learning: From Tabula Rasa to Function Approximation, CIF: Small: Online Learning and Optimal Experiment Design with a Budget, Collaborative Research: AF: Medium: Machine Learning Markets: Dynamics, Competition, and Interventions, CAREER: Toward a Foundation of Over-Parameterization, TRIPODS: Institute for Foundations of Data Science, Collaborative Research: SCALE MoDL: Adaptivity of Deep Neural Networks, Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning, AI Institute for Societal Decision Making (AI-SDM)
OpenAlex ID: W4405468215