Year: 2024
Type: preprint
Abstract: In this work, we address the problem of Hessian inversion bias in distributed second-order optimization algorithms. We introduce a novel shrinkage-based estimator for the resolvent of gram matrices which is asymptotically unbiased, and characterize its non-asymptotic convergence rate in the isotropic case. We apply this estimator to bias correction of Newton steps in distributed second-order optimization algorithms, as well as randomized sketching based methods. We examine the bias present in the naive averaging-based distributed Newton's method using analytical expressions and contrast it with our proposed bias-free approach. Our approach leads to significant improvements in convergence rate compared to standard baselines and recent proposals, as shown through experiments on both real and synthetic datasets.
Source: arXiv (Cornell University)
Authors: Zhang, Fangzhao, Pilanci, Mert
Institution: Stanford University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Demystifying Deep Machine Learning Models using Convex Optimization for Reliable AI, G1432373144, G1899085153, SenSE: Artificial Intelligence-enabled Multimodal Stress Sensing for Precision Health, Collaborative Research: Scalable Linear Algebra and Neural Network Theory, G6671297155, G7452299184
OpenAlex ID: W4391591839