Beyond the Spectral Horizon: Predicting and Accelerating Scientific Discoverability
Year: 2026
Type: article
Abstract: Building on the recently proposed zeta law of discoverability, we develop new mathematical extensions that yield a spectral theory of learning curves for finite-data machine learning. For a binary classifier scored by the area under the ROC curve, AUC(N), we expand the discriminative signal in the eigenbasis of the data covariance. Finite-sample perturbation theory implies that only eigenmodes exceeding a random-matrix detection threshold are recoverable, defining a spectral horizon K(N). The accumulated Mahalanobis information is proportional to the recoverable mutual information and, for power-law spectra, reduces to a truncated Riemann zeta sum, giving closed-form scaling laws for AUC(N). The theory predicts crossover points between competing models, staircase learning curves, spectral phase transitions, percolation-like transitions in discoverability, and when additional data, richer representations, or new sensing modalities yield the greatest gains. We then develop a stochastic extension in which AUC(N) evolves as a martingale. Malliavin calculus yields a confidence cone whose width is determined by the sensitivity of future performance to perturbations of the observed data stream. A Clark–Ocone representation reveals a duality between forward accumulation across spectral modes and backward accumulation across observations, unifying spectral and Malliavin cones within a common variational framework. Finally, we derive double-spectrum (double-zeta) laws coupling multiple covariance operators, providing a unified framework for multimodal learning, heterogeneous populations, acquisition-site effects, and domain adaptation. More broadly, the theory reframes learning as the controlled emergence of recoverable information, making the future rate of scientific discovery itself a mathematical object that can be predicted, accelerated, and optimized.
Author: THOMPSON, PAUL
Institution: University of Southern California
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204031519
FWCI:
Cites:
Cited by:
Related to: