Work

Spectral Complexity of Deep Neural Networks

Year: 2025

Type: article

Abstract: . It is well known that randomly initialized, push-forward, fully connected neural networks weakly converge to isotropic Gaussian processes in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting fields to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions and, in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

Source: SIAM Journal on Mathematics of Data Science

DOI: 10.1137/24m1675746

Authors: Simmaco Di Lillo, Domenico Marinucci, Michele Salvi, Stefano Vigogna

Institution: University of Rome Tor Vergata

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funders: Istituto Nazionale di Alta Matematica "Francesco Severi", Università degli Studi di Roma Tor Vergata, Ministero dell'Università e della Ricerca, Gruppo Nazionale per l'Analisi Matematica, la Probabilità e le loro Applicazioni

Awards: G2376554022, G2813183223, G2833937244, G750202414