Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
Year: 2020
Type: preprint
Abstract: A simple approach is proposed to obtain complexity controls for neural networks with general activation functions. The approach is motivated by approximating the general activation functions with one-dimensional ReLU networks, which reduces the problem to the complexity controls of ReLU networks. Specifically, we consider two-layer networks and deep residual networks, for which path-based norms are derived to control complexities. We also provide preliminary analyses of the function spaces induced by these norms and a priori estimates of the corresponding regularized estimators.
Source: arXiv (Cornell University)
Authors: Li, Zhong, Ma, Chao, Wu, Lei
Institutions: Peking University, Stanford University, Princeton University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W3086707176