Decaying momentum helps neural network training
Year: 2019
Type: preprint
Abstract: Momentum is a simple and popular technique in deep learning for gradient-based optimizers. We propose a decaying momentum (Demon) rule, motivated by decaying the total contribution of a gradient to all future updates. Applying Demon to Adam leads to significantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying Demon to momentum SGD improves over momentum SGD with learning rate decay in most cases. Notably, Demon momentum SGD is observed to be significantly less sensitive to parameter tuning than momentum SGD with learning rate decay schedule, critical to training deep neural networks in practice.Results are demonstrated across a variety of settings and architectures, including image classification, generative models, and language models. Demon is trivial to implement, easy to tune, and incurs limited extra computational overhead, compared to the vanilla counterparts. Code is readily available.
Source: arXiv (Cornell University)
Authors: John Chen, Anastasios Kyrillidis
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2979739940