Work

Q($λ$) with Off-Policy Corrections

Year: 2016

Type: preprint

Abstract: We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD($λ$). We illustrate this theoretical relationship empirically on a continuous-state control task.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.1602.04951

Authors: Harutyunyan, Anna, Bellemare, Marc G., Stepleton, Tom, Munos, Remi

Institutions: Vrije Universiteit Brussel, Google (United States), Google DeepMind (United Kingdom)

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: DeepMind

OpenAlex ID: W2398780562

Cites:

Cited by: 20

Related to: 20