Keyword

second-order optimization

Description: Optimization methods that use second derivatives or curvature information to guide the search for an optimum.

Primary topic: Stochastic Gradient Optimization Techniques (34%)

OpenAlex ID: keywords/second-order-optimization

On the importance of initialization and momentum in deep learning

2013 · Ilya Sutskever, James Martens, et al. · 3,739

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

2014 · Dauphin, Yann, Pascanu, Razvan, et al. · arXiv (Cornell University) · 828

PDF

Deep learning via Hessian-free optimization

2010 · James Martens · 723

HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

2019 · Zhen Dong, Zhewei Yao, et al. · IEEE/CVF International Conference on Computer Vision (ICCV) · 489

PDF

Symmetric Nonnegative Matrix Factorization for Graph Clustering

2012 · Da Kuang, Chris Ding, et al. · SIAM International Conference on Data Mining · 461

Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition

2006 · Sandhya Samarasinghe · Thammasat University Digital Collections · 448

A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning

1990 · Martin F. Møller · DAIMI Report Series · 447

PDF

Improving the convergence of back-propagation learning with second-order methods

1989 · Stephen Becker, Yann LeCun · 326

Damped Newton Algorithms for Matrix Factorization with Missing Data

2005 · A.M. Buchanan, A.W. Fitzgibbon · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · 304

New insights and perspectives on the natural gradient method

2014 · Martens, James · arXiv (Cornell University) · 283

PDF

Incorporation of Efficient Second-Order Solvers Into Latent Factor Models for Accurate Prediction of Missing QoS Data

2017 · Xin Luo, MengChu Zhou, et al. · IEEE Transactions on Cybernetics · 242

Algorithms for portfolio management based on the Newton method

2006 · Amit Agarwal, Elad Hazan, et al. · International Conference on Machine Learning · 239

Boosting Decision Trees

1995 · Harris Drucker, Corinna Cortes · neural information processing systems · 233

ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning

2021 · Zhewei Yao, Amir Gholami, et al. · Proceedings of the AAAI Conference on Artificial Intelligence · 195

PDF

Second-Order Methods for Neural Networks

1997 · Adrian J. Shepherd · Perspectives in neural computing · 180

Efficient Weight Learning for Markov Logic Networks

2007 · Daniel Lowd, Pedro Domingos · Lecture notes in computer science · 177

Loss-aware Binarization of Deep Networks

2016 · Hou, Lu, Yao, Quanming, et al. · arXiv (Cornell University) · 175

PDF

Geometry optimization with QM/MM methods II: Explicit quadratic coupling

2006 · T. Vreven, M. J. Frisch, et al. · Molecular Physics · 160

A Distributed Double-Newton Descent Algorithm for Cooperative Energy Management of Multiple Energy Bodies in Energy Internet

2020 · Yushuai Li, David Wenzhong Gao, et al. · IEEE Transactions on Industrial Informatics · 154

Three Mechanisms of Weight Decay Regularization

2018 · Zhang, Guodong, Wang, Chaoqi, et al. · arXiv (Cornell University) · 154

PDF

Finding approximate local minima faster than gradient descent

2017 · Naman Agarwal, Zeyuan Allen-Zhu, et al. · ACM SIGACT Symposium on Theory of Computing (STOC) · 152

PDF

Survey of Optimization Algorithms in Modern Neural Networks

2023 · Ruslan Abdulkadirov, Pavel Lyakhov, et al. · Mathematics · 151

PDF

Revisiting Recurrent Neural Networks for robust ASR

2012 · Oriol Vinyals, Suman V. Ravuri, et al. · IEEE International Conference on Acoustics Speech and Signal Processing · 145

RTRMC: A Riemannian trust-region method for low-rank matrix completion

2011 · Nicolas Boumal, Pierre-Antoine Absil · Digital Access to Libraries (Université catholique de Louvain (UCL), l'Université de Namur (UNamur) and the Université Saint-Louis (USL-B)) · 143

Large-scale optimal power flow: effects of initialization, decoupling and discretization

1989 · A.D. Papalexopoulos, C.F. Imparato, et al. · IEEE Transactions on Power Systems · 139

Works count: 1,345

Citation count: 20,380