Work

Bayesian Robust Optimization for Imitation Learning

Year: 2020

Type: preprint

Abstract: One of the main challenges in imitation learning is determining what action an agent should take when outside the state distribution of the demonstrations. Inverse reinforcement learning (IRL) can enable generalization to new states by learning a parameterized reward function, but these approaches still face uncertainty over the true reward function and corresponding optimal policy. Existing safe imitation learning approaches based on IRL deal with this uncertainty using a maxmin framework that optimizes a policy under the assumption of an adversarial reward function, whereas risk-neutral IRL approaches either optimize a policy for the mean or MAP reward function. While completely ignoring risk can lead to overly aggressive and unsafe policies, optimizing in a fully adversarial sense is also problematic as it can lead to overly conservative policies that perform poorly in practice. To provide a bridge between these two extremes, we propose Bayesian Robust Optimization for Imitation Learning (BROIL). BROIL leverages Bayesian reward function inference and a user specific risk tolerance to efficiently optimize a robust policy that balances expected return and conditional value at risk. Our empirical results show that BROIL provides a natural way to interpolate between return-maximizing and risk-minimizing behaviors and outperforms existing risk-sensitive and risk-neutral inverse reinforcement learning algorithms. Code is available at https://github.com/dsbrown1331/broil.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2007.12315

Authors: Brown, Daniel S., Niekum, Scott, Petrik, Marek

Institutions: University of California, Berkeley, The University of Texas at Austin, University of New Hampshire

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Office of Naval Research, Army Research Office

Awards: G1717380892, G2136314175, RI: SMALL: Robust Reinforcement Learning Using Bayesian Models, G3024042299, RI: Small: High Confidence, Efficient Learning Under Rich Task Specifications, G3881771821, CAREER: Safe and Efficient Robot Learning from Demonstration in the Real World, G608057831, NRI: Collaborative Research: Scalable Robot Autonomy through Remote Operator Assistance and Lifelong Learning, G7452299184, III: Small: Robust Reinforcement Learning for Invasive Species Management, S&AS: INT: Socially-Aware Autonomy for Long-Term Deployment of Always-On Heterogeneous Robot Teams, G8415477955, G8876996369

OpenAlex ID: W3044620414

Cites: 60

Cited by: 7

Related to: 20