Work

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Year: 2024

Type: preprint

Abstract: Let $Ω\subset \mathbb{R}^d$ be a bounded domain. We consider the problem of how efficiently shallow neural networks with the ReLU$^k$ activation function can approximate functions from Sobolev spaces $W^s(L_p(Ω))$ with error measured in the $L_q(Ω)$-norm. Utilizing the Radon transform and recent results from discrepancy theory, we provide a simple proof of nearly optimal approximation rates in a variety of cases, including when $q\leq p$, $p\geq 2$, and $s \leq k + (d+1)/2$. The rates we derive are optimal up to logarithmic factors, and significantly generalize existing results. An interesting consequence is that the adaptivity of shallow ReLU$^k$ neural networks enables them to obtain optimal approximation rates for smoothness up to order $s = k + (d+1)/2$, even though they represent piecewise polynomials of fixed degree $k$.

Source: arXiv (Cornell University)

Authors: Mao, Tong, Siegel, Jonathan W., Xu, Jinchao

Institutions: King Abdullah University of Science and Technology, Texas A&M University

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, King Abdullah University of Science and Technology, Multidisciplinary University Research Initiative, Office of Naval Research

Awards: Comparative Study of Finite Element and Neural Network Discretizations for Partial Differential Equations, G3587679958, G5121974832, G6706517093, G8876996369, CIF: Small: Interpretable Machine Learning based on Deep Neural Networks: A Source Coding Perspective

OpenAlex ID: W6979330169

Cites:

Cited by:

Related to: