Work

Distributed Adaptive Softmax Policy Gradient Method with Variance Reduction: A Global Convergence Analysis

Year: 2026

Type: article

Abstract: The distributed policy gradient (PG) method has become a mainstream technique for finding optimal policies in multi-agent reinforcement learning. However, global convergence guarantees, which are the most basic measures of theoretical convergence performance, are rarely used in existing distributed policy gradient methods. Moreover, the convergence behaviors of existing methods are sensitive to the selection of learning rates, which leads to limited convergence speeds. To tackle this challenge, we present a new distributed adaptive softmax policy gradient algorithm called DASPG by incorporating Adam-type updates and variance reductions into a vanilla distributed policy gradient method. Moreover, we establish global convergence guarantees for DASPG, i.e., an ϵ-optimal policy can be found with a sample complexity of O˜(ϵ−4.5), which is on par with leading centralized PG methods in terms of sample efficiency. Finally, we present experimental results to demonstrate the efficacy of the proposed algorithm.

Source: Mathematics

DOI: 10.3390/math14142579

Authors: Junru Shi, Huimin Gao, Qingtao Wu

Institution: Henan University of Science and Technology

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $3,137

Funder: National Natural Science Foundation of China