Work

Dimensionality Reduction, Regularization, and Generalization in Overparameterized Regressions

Year: 2022

Type: article

Abstract: Overparameterization in deep learning is powerful: Very large models fit the training data perfectly and yet often generalize well. This realization brought back the study of linear models for regression, including ordinary least squares (OLS), which, like deep learning, shows a “double-descent” behavior: (1) The risk (expected out-of-sample prediction error) can grow arbitrarily when the number of parameters $p$ approaches the number of samples $n$, and (2) the risk decreases with $p$ for $p>n$, sometimes achieving a lower value than the lowest risk for $p

Source: SIAM Journal on Mathematics of Data Science

DOI: 10.1137/20m1387821

Authors: Ningyuan Teresa, David W. Hogg, Soledad Villar

Institutions: Johns Hopkins University, Flatiron Institute

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funders: National Science Foundation, European Office of Aerospace Research and Development

Awards: G2690271170, Optimization Techniques for Geometrizing Real-World Data