Understanding the exploding gradient problem
Year: 2012
Type: preprint
Abstract: Training Recurrent Neural Networks is more troublesome than feedforward onesbecause of the vanishing and exploding gradient problems detailed in Bengio et
al. (1994). In this paper we attempt to understand the fundamental issues
underlying the exploding gradient problem by exploring it from an analytical, a
geometric and a dynamical system perspective. Our analysis is used to justify
the simple yet effective solution of norm clipping the exploded gradient. In
the experimental section, the comparison between this heuristic solution and
standard SGD provides empirical evidence towards our hypothesis as well as it
shows that such a heuristic is required to reach state of the art results on a
character prediction task and a polyphonic music prediction one.
Source: arXiv (Cornell University)
Authors: Razvan Pascanu, Tomáš Mikolov, Yoshua Bengio
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2402302915