Work

Identifiability in inverse reinforcement learning

Year: 2021

Type: conference-paper

Abstract: Inverse reinforcement learning attempts to reconstruct the reward function in a Markov decision problem, using observations of agent actions. As already observed in Russell [1998] the problem is ill-posed, and the reward function is not identifiable, even under the presence of perfect information about optimal behavior. We provide a resolution to this non-identifiability for problems with entropy regularization. For a given environment, we fully characterize the reward functions leading to a given policy and demonstrate that, given demonstrations of actions for the same reward under two distinct discount factors, or under sufficiently different environments, the unobserved reward can be recovered up to a constant. We also give general necessary and sufficient conditions for reconstruction of time-homogeneous rewards on finite horizons, and for action-independent rewards, generalizing recent results of Kim et al. [2021] and Fu et al. [2018].

Source: Oxford University Research Archive (ORA) (University of Oxford)

DOI: 10.48550/arxiv.2106.03498

Authors: Cao, Haoyang, Cohen, Samuel N., Szpruch, Lukasz

Institutions: The Alan Turing Institute, University of California, Berkeley, University of Oxford, University College London, University of Edinburgh

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: University of Oxford, Engineering and Physical Sciences Research Council

Awards: G1633672591, G444630160, The Alan Turing Institute, G7723329525, G8597009950