Work

Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis

Year: 2021

Type: article

Abstract: We address the problem of policy evaluation in discounted, tabular Markov decision processes and provide instance-dependent guarantees on the $\ell_\infty$-error under a generative model. We establish both asymptotic and nonasymptotic versions of local minimax lower bounds for policy evaluation, thereby providing an instance-dependent baseline by which to compare algorithms. Theory-inspired simulations show that the widely used temporal difference (TD) algorithm is strictly suboptimal when evaluated in a nonasymptotic setting, even when combined with Polyak--Ruppert iterate averaging. We remedy this issue by introducing and analyzing variance-reduced forms of stochastic approximation, showing that they achieve nonasymptotic, instance-dependent optimality up to logarithmic factors.

Source: SIAM Journal on Mathematics of Data Science

DOI: 10.1137/20m1331524

Authors: Koulik Khamaru, Ashwin Pananjady, Feng Ruan, Martin J. Wainwright, Michael I. Jordan

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funders: National Science Foundation, U.S. Department of Defense, Defense Advanced Research Projects Agency, Office of Naval Research

Awards: G1238194512, G1717380892, G2520373344, G4353487559, Statistical Estimation in Resource-Constrained Environments: Computation, Communication and Privacy, Iterative Algorithms for Statistics: From Convergence Rates to Statistical Accuracy, G8876996369

OpenAlex ID: W3012451757

FWCI: 2.029

Cites: 52

Cited by: 37

Related to: 10