Year: 2022
Type: preprint
Abstract: This work theoretically studies stochastic neural networks, a main type of neural network in use. We prove that as the width of an optimized stochastic neural network tends to infinity, its predictive variance on the training set decreases to zero. Our theory justifies the common intuition that adding stochasticity to the model can help regularize the model by introducing an averaging effect. Two common examples that our theory can be relevant to are neural networks with dropout and Bayesian latent variable models in a special limit. Our result thus helps better understand how stochasticity affects the learning of neural networks and potentially design better architectures for practical problems.
Source: arXiv (Cornell University)
Authors: Ziyin, Liu, Zhang, Hanlin, Meng, Xiangming, Lu, Yuting, Xing, Eric, Ueda, Masahito
Institutions: The University of Tokyo, Carnegie Mellon University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4221163263