Work

Exploring and Addressing Reward Confusion in Offline Preference Learning

Year: 2024

Type: preprint

Abstract: Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2407.16025

Authors: Chen, Xin, Toyer, Sam, Shkurti, Florian

Institutions: ETH Zurich, University of California, Berkeley, University of Toronto

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W6891964254

Cites:

Cited by:

Related to: