Year: 2025
Type: article
Abstract: A fundamental problem in machine learning is to understand how neural networks make accurate predictions, while seemingly bypassing the curse of dimensionality. A possible explanation is that common training algorithms for neural networks implicitly perform dimensionality reduction—a process called feature learning. Recent work [A. Radhakrishnan, D. Beaglehole, P. Pandit, M. Belkin, Science 383 , 1461–1467 (2024).] posited that the effects of feature learning can be elicited from a classical statistical estimator called the average gradient outer product (AGOP). The authors proposed Recursive Feature Machines (RFMs) as an algorithm that explicitly performs feature learning by alternating between 1) reweighting the feature vectors by the AGOP and 2) learning the prediction function in the transformed space. In this work, we develop theoretical guarantees for how RFM performs dimensionality reduction by focusing on the class of overparameterized problems arising in sparse linear regression and low-rank matrix recovery. Specifically, we show that RFM restricted to linear models (lin-RFM) reduces to a variant of the well-studied Iteratively Reweighted Least Squares (IRLS) algorithm. Furthermore, our results connect feature learning in neural networks and classical sparse recovery algorithms and shed light on how neural networks recover low rank structure from data. In addition, we provide an implementation of lin-RFM that scales to matrices with millions of missing entries. Our implementation is faster than the standard IRLS algorithms since it avoids forming singular value decompositions. It also outperforms deep linear networks for sparse linear regression and low-rank matrix completion.
Language: English
Subfield: Computational Mechanics
Field: Engineering
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1353381076, G1453181110, G1503004319, G1523888516, TRIPODS: Algorithms for Data Science: Complexity, Scalability, and Robustness., Track 4: Advanced CI Coordination Ecosystem: Monitoring and Measurement Services, Exploiting Smooth Substructure in Non-Smooth Stochastic Optimization, G2114340465, G2647087871, G298330452, G3226045097, G3562386289, Collaboration on the Theoretical Foundations of Deep Learning, XSEDE 2.0: Integrating, Enabling and Enhancing National Cyberinfrastructure with Expanding Community Involvement, AI Institute for Learning-Enabled Optimization at Scale (TILOS), Track 1: ACCESS Resource Allocations Marketplace and Platform Services (RAMPS), G4010157413, G4403015908, ACO: An Open CI Ecosystem to Advance Scientific Discovery (OpenCI), G4875486994, Track 3: COre National Ecosystem for CyberinfrasTructure (CONECT), G5143113707, G5367127336, G5464919783, G5803504232, TRIPODS: Institute for Foundations of Data Science, G6098521345, G8108599699, Track 2: Customized Multi-tier Assistance, Training, and Computational Help (MATCH) for End User ACCESS to CI, G8505604578
OpenAlex ID: W4408928873