Year: 2022
Type: preprint
Abstract: In this work, we consider and analyze the sample complexity of model-free reinforcement learning with a generative model. Particularly, we analyze mirror descent value iteration (MDVI) by Geist et al. (2019) and Vieillard et al. (2020a), which uses the Kullback-Leibler divergence and entropy regularization in its value and policy updates. Our analysis shows that it is nearly minimax-optimal for finding an $\varepsilon$-optimal policy when $\varepsilon$ is sufficiently small. This is the first theoretical result that demonstrates that a simple model-free algorithm without variance-reduction can be nearly minimax-optimal under the considered setting.
Source: arXiv (Cornell University)
Institutions: University of Alberta, Peking University, Google (United States), Centre National de la Recherche Scientifique, Institut national de recherche en sciences et technologies du numérique, Institut Élie Cartan de Lorraine, Université de Lorraine, The University of Tokyo, Otto-von-Guericke-Universität Magdeburg
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4282041773