Work

PGDOT -- Perturbed Gradient Descent Adapted with Occupation Time

Year: 2020

Type: preprint

Abstract: This paper develops further the idea of perturbed gradient descent (PGD), by adapting perturbation with the history of states via the notion of occupation time. The proposed algorithm, perturbed gradient descent adapted with occupation time (PGDOT), is shown to converge at least as fast as the PGD algorithm and is guaranteed to avoid getting stuck at saddle points. The analysis is corroborated by empirical studies, in which a mini-batch version of PGDOT is shown to outperform alternatives such as mini-batch gradient descent, Adam, AMSGrad, and RMSProp in training multilayer perceptrons (MLPs). In particular, the mini-batch PGDOT manages to escape saddle points whereas these alternatives fail.

Source: arXiv (Cornell University)

Authors: Xin Guo, Jiequn Han, Mahan Tajrobehkar, Wenpin Tang

Institutions: University of California, Berkeley, Princeton University, Columbia University

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W3171826361

Cites: 60

Cited by: 4

Related to: 10