Work

A2C is a special case of PPO

Year: 2022

Type: preprint

Abstract: Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using \texttt{Stable-baselines3}, showing A2C and PPO produce the \textit{exact} same models when other settings are controlled.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2205.09123

Authors: Huang, Shengyi, Kanervisto, Anssi, Raffin, Antonin, Wang, Weixun, Ontañón, Santiago, Dossa, Rousslan Fernand Julien

Institutions: Drexel University, University of Eastern Finland, Deutsches Zentrum für Luft- und Raumfahrt e. V. (DLR), Tianjin University

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4281250801

Cites: 2

Cited by: 34

Related to: 10