Work

The Dynamics of Spectral Bias in Operator Learning

Year: 2026
Type: preprint
Abstract: Spectral bias is the phenomenon where neural networks learn the low-wavenumber content of a target before its high-wavenumber content. We analyze the training dynamics of neural operators on the discrete Poisson problem and find that the bias is a property of the eigenbasis of the kernel that governs the training update, not of the wavenumber basis in which it is usually reported. A model learning a discretized operator retains its error in the eigendirections of this kernel with small eigenvalues, which are the rates at which training contracts the error along those directions. Under smooth forcing, low variance is placed at high wavenumbers, and any model, linear or nonlinear, has small eigenvalues along the eigenvectors with high-wavenumber content. Under white forcing, the linear models learn the operator, while the nonlinear models retain error in slow directions that depend on the coupling between training samples in the kernel; for the GELU dense network they disappear entirely when that coupling is removed. Modes that a model cannot represent at all contribute zero eigenvalues. The slow eigenvectors decide which wavenumbers carry this error, through their projection onto the sine modes. For the spectral operator and the linear Fourier neural operator these eigenvectors are close to sine modes, so each wavenumber has its own rate and the error collects at high wavenumbers. Delocalized models, such as the dense networks under either forcing, have error of nearly the same magnitude at every wavenumber. The spectrum of the true solution decays with $k$ and falls below this floor at some wavenumber, past which the relative error grows. No mode is learned more slowly than another in this case; the growth comes from the decay of the target spectrum. Spectral bias is usually reported by looking at this retained error in the sine basis, and we use its projection onto the kernel's eigendirections to explain the shape it has there.
Language: English
Open Access status: green
OpenAlex ID: W7226022344
Cites: 2
Cited by:
Related to: 10