Year: 2026
Type: preprint
Abstract: Solving data-driven problems requires defining complex models and fitting them to data, neural networks being a motivating example. The fitting procedure can be seen as an optimization problem, which is often non-convex, and hence optimization guarantees are hard to derive. An opportunity is provided by viewing the model of interest as a redundant reparameterization--an overparameterization--of some simpler model for which optimization results are easier to achieve. In this paper, after formalizing the above idea, we revisit some recent results and derive new ones. In particular, we consider the gradient flow of some classes of linear overparameterizations and show that they correspond to suitable mirror flows on the original parameters. Our main contribution relates to the study of the latter, for which we establish well-posedness and convergence. Several specific instances are provided, including fully connected linear networks and networks with weight normalization. The results yield insight into the role of overparameterization in optimization and corresponding implicit regularization properties.
Source: arXiv (Cornell University)
Institutions: Universidad Técnica Federico Santa María, University of Genoa
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, Istituto Nazionale di Alta Matematica "Francesco Severi", European Commission, Ministero dell’Istruzione, dell’Università e della Ricerca, Universidad de Tarapacá, Companhia Brasileira de Metalurgia e Mineração, Horizon 2020 Framework Programme, Gruppo Nazionale per l'Analisi Matematica, la Probabilità e le loro Applicazioni, Air Force Office of Scientific Research
Awards: G1282966504, G2510020596, Nonlocal Methods for Arbitrary Data Sources, G3045592441, G3345895242, G3477743816, G3479177607, G3503992018, G4149313689, G480903921, G5096011677, G5622661325, A Center for Brains, Minds and Machines: the Science and the Technology of Intelligence, Training Data-driven Experts in OPTimization, G7809643752
OpenAlex ID: W7221020546