High-Probability Sample Complexities for Policy Evaluation With Linear Function Approximation
Year: 2024
Type: article
Abstract: This paper is concerned with the problem of policy evaluation with linear function approximation in discounted infinite horizon Markov decision processes. We investigate the sample complexities required to guarantee a predefined estimation error of the best linear coefficients for two widely-used policy evaluation algorithms: the temporal difference (TD) learning algorithm and the two-timescale linear TD with gradient correction (TDC) algorithm. In both the on-policy setting, where observations are generated from the target policy, and the off-policy setting, where samples are drawn from a behavior policy potentially different from the target policy, we establish the first sample complexity bound with high-probability convergence guarantee that attains the optimal dependence on the tolerance level. We also exhibit an explicit dependence on problem-related quantities, and show in the on-policy setting that our upper bound matches the minimax lower bound on crucial problem parameters, including the choice of the feature map and the problem dimension.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, Princeton University, University of Pennsylvania, University of Chicago, Carnegie Mellon University, Tsinghua University, Chinese University of Hong Kong, Peking University, Institut Carnot Santé Animale, National Institutes of Health, Wharton School, University of Pennsylvania, Office of Naval Research
Awards: Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality, G1469985387, CAREER: Statistical Learning from a Modern Perspective: Over-parameterization, Regularization, and Generalization, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, Taming distribution shifts in statistical learning, G6671297155, Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality, G8876996369
OpenAlex ID: W4396215051