Unified finite-time error analysis of soft Q-learning
Year: 2025
Type: article
Abstract: Soft Q-learning is one of the most commonly used reinforcement learning algorithms for various purposes, e.g., dealing with entropy-regularized Markov decision problems, reducing the overestimation bias, and improving explorations. Its effectiveness in practice has led to its widespread use; however, there has not been much theoretical study on soft Q-learning. This paper attempts to provide an integrated finite-time analytical approach for soft Q-learning from a control-theoretic perspective. We examine three different kinds of soft Q-learning algorithms that use the log-sum-exp operator, the Boltzmann operator, and the mellowmax operator, respectively. Utilizing dynamical switching system models, we obtain the finite-time error bounds of three soft Q-learning variants. We believe that our analysis can assist in a better understanding of soft Q-learning through links with switching system models.
Source: Neurocomputing
Authors: Narim Jeong, Donghwan Lee
Institution: Korea Advanced Institute of Science and Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,670
OpenAlex ID: W4407147140