Work

Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors

Year: 1993
Type: book-chapter
Abstract: We propose a very simple, and well principled wayofcomputing the optimal step size in gradient descent algorithms. The on-line version is very efficient computationally, and is applicable to large backpropagation networks trained on large data sets. The main ingredient is a technique for estimating the principal eigenvalue(s) and eigenvector(s) of the objective function's second derivativematrix (Hessian), which does not require to even calculate the Hessian. Several other applications of this technique are proposed for speeding up learning, or for eliminating useless parameters.
Language: English
Open Access status: green
OpenAlex ID: W2142426721
FWCI: 2.38
Cites: 8
Cited by: 79
Related to: 10