Work

PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration

Year: 2022

Type: preprint

Abstract: Proximal Policy Optimization (PPO) is a highly popular policy-based deep reinforcement learning (DRL) approach. However, we observe that the homogeneous exploration process in PPO could cause an unexpected stability issue in the training phase. To address this issue, we propose PPO-UE, a PPO variant equipped with self-adaptive uncertainty-aware explorations (UEs) based on a ratio uncertainty level. The proposed PPO-UE is designed to improve convergence speed and performance with an optimized ratio uncertainty level. Through extensive sensitivity analysis by varying the ratio uncertainty level, our proposed PPO-UE considerably outperforms the baseline PPO in Roboschool continuous control tasks.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2212.06343

Authors: Zhang, Qisheng, Guo, Zhen, Jøsang, Audun, Kaplan, Lance M., Chen, Feng, Jeong, Dong H., Cho, Jin-Hee

Institutions: Virginia Tech, University of Oslo, United States Department of the Army, DEVCOM Army Research Laboratory, The University of Texas at Dallas, Jackson State University, Jackson College, University of the District of Columbia

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Army Research Office

Awards: III: Medium: Collaborative Research: MUDL: Multidimensional Uncertainty-Aware Deep Learning Framework, III: Medium: Collaborative Research: MUDL: Multidimensional Uncertainty-Aware Deep Learning Framework, G5479568110, III: Medium: Collaborative Research: MUDL: Multidimensional Uncertainty-Aware Deep Learning Framework, G8876858818

OpenAlex ID: W4311554115

Cites:

Cited by:

Related to: 10