Work

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Year: 2016

Type: preprint

Abstract: We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.1605.07154

Authors: Neyshabur, Behnam, Wu, Yuhuai, Salakhutdinov, Ruslan, Srebro, Nathan

Institutions: Toyota Technological Institute at Chicago, University of Toronto, Carnegie Mellon University, University of Chicago

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Intel Collaboration Research Institute for Computational Intelligence

OpenAlex ID: W2401137308

Cites: 12

Cited by: 19

Related to: 20