Work

Deep Learning Meets Sparse Regularization: A signal processing perspective

Year: 2023

Type: article

Abstract: Deep learning (DL) has been wildly successful in practice, and most of the state-of-the-art machine learning methods are based on neural networks (NNs). Lacking, however, is a rigorous mathematical theory that adequately explains the amazing performance of deep NNs (DNNs). In this article, we present a relatively new mathematical framework that provides the beginning of a deeper understanding of DL. This framework precisely characterizes the functional properties of NNs that are trained to fit to data. The key mathematical tools that support this framework include transform-domain sparse regularization, the Radon transform of computed tomography, and approximation theory, which are all techniques deeply rooted in signal processing. This framework explains the effect of weight decay regularization in NN training, use of skip connections and low-rank weight matrices in network architectures, role of sparsity in NNs, and explains why NNs can perform well in high-dimensional problems.

Source: IEEE Signal Processing Magazine

DOI: 10.1109/msp.2023.3286988

Authors: Rahul Parhi, Robert D. Nowak

Institutions: École Polytechnique Fédérale de Lausanne, University of Wisconsin–Madison

Language: English

Topic: Sparse and Compressive Sensing Techniques

Subfield: Computational Mechanics

Field: Engineering

Domain: Physical Sciences

Open Access status: green