Year: 2023
Type: report
Abstract: In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for multilayered sparse ReLU neural networks, including convolutional neural networks. These bounds differ from previous ones, as they consider the norms of the convolutional filters instead of the norms of the associated Toeplitz matrices, independently of weight sharing between neurons. As we show theoretically, these bounds may be orders of magnitude better than standard norm- based generalization bounds and empirically, they are almost non-vacuous in estimating generalization in various simple classification problems. Taken together, these results suggest that compositional sparsity of the underlying target function is critical to the success of deep neural networks.
Institutions: Massachusetts Institute of Technology, Tel Aviv University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Proposal: Object and Action Recognition in Time Sequences of Images: Computational Neuroscience and Neurophysiology, Computational Models and Physiological Studies of Feedback in Visual Object Recognition Tasks, A Center for Brains, Minds and Machines: the Science and the Technology of Intelligence, G7809643752
OpenAlex ID: W7021247103