Work

Infinite‐width limit of deep linear neural networks

Year: 2024

Type: article

Abstract: This paper studies the infinite‐width limit of deep linear neural networks (NNs) initialized with random parameters. We obtain that, when the number of parameters diverges, the training dynamics converge (in a precise sense) to the dynamics obtained from a gradient descent on an infinitely wide deterministic linear NN. Moreover, even if the weights remain random, we get their precise law along the training dynamics, and prove a quantitative convergence result of the linear predictor in terms of the number of parameters. We finally study the continuous‐time limit obtained for infinitely wide linear NNs and show that the linear predictors of the NN converge at an exponential rate to the minimal ‐norm minimizer of the risk.

Source: Communications on Pure and Applied Mathematics

DOI: 10.1002/cpa.22200

Authors: Lénaïc Chizat, Maria Colombo, Xavier Fernández‐Real, Alessio Figalli

Institutions: École Polytechnique Fédérale de Lausanne, ETH Zurich

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $4,980

Funders: Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Stavros Niarchos Foundation, European Research Council, Agencia Estatal de Investigación