Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
Year: 2023
Type: article
Abstract: . Policy optimization, which learns the policy of interest by maximizing the value function via large-scale optimization techniques, lies at the heart of modern reinforcement learning (RL). In addition to value maximization, other practical considerations arise commonly as well, including the need of encouraging exploration, and that of ensuring certain structural properties of the learned policy due to safety, resource, and operational constraints. These considerations can often be accounted for by resorting to regularized RL, which augments the target value function with a structure-promoting regularization term. Focusing on an infinite-horizon discounted tabular Markov decision process, this paper proposes a generalized policy mirror descent (GPMD) algorithm for solving regularized RL. As a generalization of policy mirror descent [G. Lan, Math. Program., 198 (2023), pp. 1059–1106], the proposed algorithm accommodates a general class of convex regularizers as well as a broad family of Bregman divergence in cognizance of the regularizer in use. We demonstrate that our algorithm converges linearly to the global solution over an entire range of learning rates, in a dimension-free fashion, even when the regularizer lacks strong convexity and smoothness. In addition, this linear convergence feature is provably stable in the face of inexact policy evaluation and imperfect policy updates. Numerical experiments are provided to corroborate the applicability and appealing performance of GPMD.
Source: SIAM Journal on Optimization
DOI: 10.1137/21m1456789
Authors: Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D. Lee, Yuejie Chi
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1056225131, G1469985387, G1523888516, RINGS: Learning-Enabled Ground and Air Integrated Networks (GAINs), CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks, G2376053866, G2384362821, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, G4353487559, G4720003262, RI: Small: Uncertainty Quantification for Nonconvex Low-Complexity Models, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, CIF: Small: Taming Nonconvexity in High-Dimensional Statistical Estimation, G5972442391, G6098521345, CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks, CIF: Small: Resource-Efficient Statistical Inference in Networked Environments, G7452299184, RI: Medium: Collaborative Research:Algorithmic High-Dimensional Statistics: Optimality, Computtional Barriers, and High-Dimensional Corrections, G862988727, RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks, G8876996369, Collaborative Research: Towards a Theoretic Foundation for Optimal Deep Graph Learning
OpenAlex ID: W4381739798