Year: 2025
Type: preprint
Abstract: Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.
Source: arXiv (Cornell University)
Authors: Dangel, Felix, Eschenhagen, Runa, Ormaniec, Weronika, Fernandez, Andres, Tatzel, Lukas, Kristiadi, Agustinus
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: Canadian Institute for Advanced Research, International Max Planck Research School for Environmental, Cellular and Molecular Microbiology, Government of Canada, European Commission, Deutsche Forschungsgemeinschaft, Bundesministerium für Bildung und Forschung, Cambridge Trust, International Max Planck Research School for Advanced Methods in Process and Systems Engineering
OpenAlex ID: W4407091379