Year: 2024
Type: preprint
Abstract: Quadratic approximations form a fundamental building block of machine learning methods. E.g., second-order optimizers try to find the Newton step into the minimum of a local quadratic proxy to the objective function; and the second-order approximation of a network's loss function can be used to quantify the uncertainty of its outputs via the Laplace approximation. When computations on the entire training set are intractable - typical for deep learning - the relevant quantities are computed on mini-batches. This, however, distorts and biases the shape of the associated stochastic quadratic approximations in an intricate way with detrimental effects on applications. In this paper, we (i) show that this bias introduces a systematic error, (ii) provide a theoretical explanation for it, (iii) explain its relevance for second-order optimization and uncertainty quantification via the Laplace approximation in deep learning, and (iv) develop and evaluate debiasing strategies.
Source: arXiv (Cornell University)
Institutions: Tübingen AI Center, University of Tübingen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: International Max Planck Research School for Environmental, Cellular and Molecular Microbiology, European Commission, Deutsche Forschungsgemeinschaft, Bundesministerium für Bildung und Forschung, International Max Planck Research School for Advanced Methods in Process and Systems Engineering, HORIZON EUROPE European Research Council
OpenAlex ID: W4403995744