Year: 2021
Type: preprint
Abstract: To understand how deep learning works, it is crucial to understand the training dynamics of neural networks. Several interesting hypotheses about these dynamics have been made based on empirically observed phenomena, but there exists a limited theoretical understanding of when and why such phenomena occur. In this paper, we consider the training dynamics of gradient flow on kernel least-squares objectives, which is a limiting dynamics of SGD trained neural networks. Using precise high-dimensional asymptotics, we characterize the dynamics of the fitted model in two "worlds": in the Oracle World the model is trained on the population distribution and in the Empirical World the model is trained on a sampled dataset. We show that under mild conditions on the kernel and $L^2$ target regression function the training dynamics undergo three stages characterized by the behaviors of the models in the two worlds. Our theoretical results also mathematically formalize some interesting deep learning phenomena. Specifically, in our setting we show that SGD progressively learns more complex functions and that there is a "deep bootstrap" phenomenon: during the second stage, the test error of both worlds remain close despite the empirical training error being much smaller. Finally, we give a concrete example comparing the dynamics of two different kernels which shows that faster training is not necessary for better generalization.
Source: arXiv (Cornell University)
Authors: Ghosh, Nikhil, Mei, Song, Yu, Bin
Institution: University of California, Berkeley
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: TRIPODS: Berkeley Institute on the Foundations of Data Analysis, G2647087871, Parallel Ensemble Learning and Feature Interaction Discovery: High Volume Dynamic Data, Collaboration on the Theoretical Foundations of Deep Learning, Understanding Complexity and the Bias-Variance Tradeoff in High Dimensions: Theory and Data Evidence, Canonical Linear Methods and Hierarchical Non-Linear Methods in High-Dimensional Statistics, Emerging Frontiers of Science of Information, Foundations of Data Science Institute, BIGDATA: F: Scalable and Interpretable Machine Learning: Bridging Mechanistic and Data-Driven Modeling in the Biological Sciences
OpenAlex ID: W3213989869