Year: 2022
Type: preprint
Abstract: Learning to differentiate model distributions from observed data is a fundamental problem in statistics and machine learning, and high-dimensional data remains a challenging setting for such problems. Metrics that quantify the disparity in probability distributions, such as the Stein discrepancy, play an important role in high-dimensional statistical testing. In this paper, we investigate the role of $L^2$ regularization in training a neural network Stein critic so as to distinguish between data sampled from an unknown probability distribution and a nominal model distribution. Making a connection to the Neural Tangent Kernel (NTK) theory, we develop a novel staging procedure for the weight of regularization over training time, which leverages the advantages of highly-regularized training at early times. Theoretically, we prove the approximation of the training dynamic by the kernel optimization, namely the ``lazy training'', when the $L^2$ regularization weight is large, and training on $n$ samples converge at a rate of ${O}(n^{-1/2})$ up to a log factor. The result guarantees learning the optimal critic assuming sufficient alignment with the leading eigen-modes of the zero-time NTK. The benefit of the staged $L^2$ regularization is demonstrated on simulated high dimensional data and an application to evaluating generative models of image data.
Source: arXiv (Cornell University)
Authors: Repasky, Matthew, Cheng, Xiuyuan, Xie, Yao
Institutions: Georgia Institute of Technology, Duke University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: ATD: Scanning Dynamic Spatial-Temporal Discrete Events for Threat Detection, Bridging Statistical Hypothesis Tests and Deep Learning for Reliability and Computational Efficiency, CAREER: Quick Detection for Streaming Data Over Dynamic Networks, G3387330441, G3986188589, G4324459455, Data-Driven Optimal Police Patrol Zone Districting and Staffing, Sequential Detection and Prediction for Solar Situation Awareness in Power Networks, AI Institute for Advances in Optimization, G6127310570, G6671297155, G7238632903, G7911581280, G7976766739, Collaborative Research: Geometric Analysis and Computation for Generative Models, G975068223
OpenAlex ID: W4284973272