A Library of Mirrors: Deep Neural Nets in Low Dimensions Are Convex Lasso Models with Reflection Features
Year: 2026
Type: article
Abstract: . We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks with piecewise linear activations and depths ranging from 2 to an arbitrary but finite number of layers. We first show that 2-layer networks with piecewise linear activations are equivalent to Lasso models using a discrete dictionary of ramp functions, with breakpoints corresponding to the training data points. In certain general architectures with absolute value or ReLU activations, a third layer surprisingly creates features that reflect the training data about themselves. Additional layers progressively generate reflections of these reflections. The Lasso representation provides valuable insights into the analysis of globally optimal networks, elucidating their solution landscapes and enabling closed-form solutions in certain minimal regularization cases. Numerical results show that reflections also occur when optimizing standard deep networks using standard nonconvex optimizers. Additionally, we demonstrate our theory with autoregressive time-series models.
DOI: 10.1137/24m1707845
Institution: Stanford University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Funder: National Science Foundation
OpenAlex ID: W4392490424