Year: 2024
Type: preprint
Abstract: Single-Index Models are high-dimensional regression problems with planted structure, whereby labels depend on an unknown one-dimensional projection of the input via a generic, non-linear, and potentially non-deterministic transformation. As such, they encompass a broad class of statistical inference tasks, and provide a rich template to study statistical and computational trade-offs in the high-dimensional regime. While the information-theoretic sample complexity to recover the hidden direction is linear in the dimension $d$, we show that computationally efficient algorithms, both within the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) framework, necessarily require $Ω(d^{k^\star/2})$ samples, where $k^\star$ is a "generative" exponent associated with the model that we explicitly characterize. Moreover, we show that this sample complexity is also sufficient, by establishing matching upper bounds using a partial-trace algorithm. Therefore, our results provide evidence of a sharp computational-to-statistical gap (under both the SQ and LDP class) whenever $k^\star>2$. To complete the study, we provide examples of smooth and Lipschitz deterministic target functions with arbitrarily large generative exponents $k^\star$.
Source: arXiv (Cornell University)
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning, RI:Small:NSF-BSF: Computational and Statistical Tradeoffs in Inverse Problems using Deep Learning, G2384362821, CAREER: Towards a Theory of Deep Learning, CHS: Medium: Geometric Deep Learning for Accurate and Efficient Physics Simulation, CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks, G6671297155, G7452299184, Statistical and Computational Foundations of Deep Generative Models, CAREER: CIF: Theory and Applications of Geometric Deep Learning, G862988727, RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks
OpenAlex ID: W4392679430