Year: 2023
Type: article
Abstract: This paper investigates a model-free algorithm of broad interest in reinforcement learning, namely, Q-learning. Whereas substantial progress had been made toward understanding the sample efficiency of Q-learning in recent years, it remained largely unclear whether Q-learning is sample-optimal and how to sharpen the sample complexity analysis of Q-learning. In this paper, we settle these questions: (1) When there is only a single action, we show that Q-learning (or, equivalently, TD learning) is provably minimax optimal. (2) When there are at least two actions, our theory unveils the strict suboptimality of Q-learning and rigorizes the negative impact of overestimation in Q-learning. Our theory accommodates both the synchronous case (i.e., the case in which independent samples are drawn) and the asynchronous case (i.e., the case in which one only has access to a single Markovian trajectory).
Source: Operations Research
Institutions: University of Pennsylvania, Carnegie Mellon University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality, G1056225131, G1469985387, G1523888516, G1717380892, Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality, G2376053866, CAREER: Statistical Learning from a Modern Perspective: Over-parameterization, Regularization, and Generalization, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, Taming Nonlinear Inverse Problems: Theory and Algorithms, G4353487559, G4720003262, RI: Small: Uncertainty Quantification for Nonconvex Low-Complexity Models, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, CIF: Small: Taming Nonconvexity in High-Dimensional Statistical Estimation, G5972442391, G6098521345, CIF: Small: Resource-Efficient Statistical Inference in Networked Environments, G6671297155, G6691530897, Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality, RI: Medium: Collaborative Research:Algorithmic High-Dimensional Statistics: Optimality, Computtional Barriers, and High-Dimensional Corrections, CIF: Medium: Collaborative Research: Nonconvex Optimization for High-Dimensional Signal Estimation: Theory and Fast Algorithms, G854152385, G8876996369, Collaborative Research: Towards a Theoretic Foundation for Optimal Deep Graph Learning
OpenAlex ID: W4366692289