A study of the Exponentiated Gradient +/- algorithm for stochastic optimization of neural networks
Year: 2019
Type: dissertation
Abstract: Exponentiated Gradient +/- (abbr. EG+-) is a gradient update algorithmdrawn from work by Manfred Warmuth (Kivinen and Warmuth, 1997) in theonline learning setting. This thesis ports the algorithm into the context of deepneural networks and analyses its fitness in that context compared to the currentstate of the art gradient update methods. Existing methods employ an additiveupdate scheme whereby some fraction of the gradient is added to the weightvalues to update them at each iteration in the gradient descent algorithm. EG+-provides a multiplicative update scheme whereby a proportion of the gradientis multiplied into the original weight value, and then normalized to update theweight. EG+- is motivated by using a relative entropy regularization. This thesisanalyzes various properties and experimental results of the algorithm in comparisonto other update methods, and analyzes EG+- in the context of state of theart residual networks and challenging vision problems. Three published implementationsare experimented with, and demonstrate that EG+- performs betterthan SGD when there are many noisy features, and that it compares well withcommonly used state-of-the art gradient descent optimization methods. EG+-also performs better than most SGD based optimizers on black-box adversarialattacks, with the exception of non momentum based SGD with which it performssimilarly.
Author: David Freeman Parks
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2980032880