Year: 2023
Type: preprint
Abstract: Policy optimization methods with function approximation are widely used in multi-agent reinforcement learning. However, it remains elusive how to design such algorithms with statistical guarantees. Leveraging a multi-agent performance difference lemma that characterizes the landscape of multi-agent policy optimization, we find that the localized action value function serves as an ideal descent direction for each local policy. Motivated by the observation, we present a multi-agent PPO algorithm in which the local policy of each agent is updated similarly to vanilla PPO. We prove that with standard regularity conditions on the Markov game and problem-dependent quantities, our algorithm converges to the globally optimal policy at a sublinear rate. We extend our algorithm to the off-policy setting and introduce pessimism to policy evaluation, which aligns with experiments. To our knowledge, this is the first provably convergent multi-agent PPO algorithm in cooperative Markov games.
Source: arXiv (Cornell University)
Institutions: Princeton University, Yale University, Northwestern University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning, G2384362821, CAREER: Towards a Theory of Deep Learning, CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks, G6671297155, G7452299184, G862988727, RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks
OpenAlex ID: W4375959468