Year: 2024
Type: preprint
Abstract: Transfer learning is a critical part of real-world machine learning deployments and has been extensively studied in experimental works with overparameterized neural networks. However, even in the simplest setting of linear regression a notable gap still exists in the theoretical understanding of transfer learning. In-distribution research on high-dimensional linear regression has led to the identification of a phenomenon known as \textit{benign overfitting}, in which linear interpolators overfit to noisy training labels and yet still generalize well. This behavior occurs under specific conditions on the source covariance matrix and input data dimension. Therefore, it is natural to wonder how such high-dimensional linear models behave under transfer learning. We prove the first non-asymptotic excess risk bounds for benignly-overfit linear interpolators in the transfer learning setting. From our analysis, we propose a taxonomy of \textit{beneficial} and \textit{malignant} covariate shifts based on the degree of overparameterization. We follow our analysis with empirical studies that show these beneficial and malignant covariate shifts for linear interpolators on real image data, and for fully-connected neural networks in settings where the input data dimension is larger than the training sample size.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1141332560, G1353381076, G1453181110, G1503004319, Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, G2647087871, G298330452, G3562386289, Collaboration on the Theoretical Foundations of Deep Learning, Understanding Complexity and the Bias-Variance Tradeoff in High Dimensions: Theory and Data Evidence, Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), G4010157413, G4403015908, Advancing Theory and Methodology for Tree-Based Algorithms in High Dimensions, ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), G5367127336, G5464919783, Foundations of Data Science Institute, RTG: Advancing Machine Learning - Causality and Interpretability, G8108599699, Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI
OpenAlex ID: W4393904889