Work

Prox-SGD: Training Structured Neural Networks under Regularization and Constraints

Year: 2020

Type: conference-paper

Abstract: In this paper, we consider the problem of training structured neural networks (NN) with nonsmooth regularization (e.g.ℓ 1 -norm) and constraints (e.g.interval constraints).We formulate training as a constrained nonsmooth nonconvex optimization problem, and propose a convergent proximal-type stochastic gradient descent (ProxSGD) algorithm.We show that under properly selected learning rates, with probability 1, every limit point of the sequence generated by the proposed Prox-SGD algorithm is a stationary point.Finally, to support the theoretical analysis and demonstrate the flexibility of ProxSGD, we show by extensive numerical tests how ProxSGD can be used to train either sparse or binary neural networks through an adequate selection of the regularization function and constraint set.

Source: Open Repository and Bibliography (University of Luxembourg)

Authors: Yang Yang, Yaxiong Yuan, Avraam Chatzimichailidis, Ruud J. G. van Sloun, Lei Lei, Symeon Chatzinotas

Institutions: Fraunhofer-Gesellschaft, University of Luxembourg, Eindhoven University of Technology

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Deutsche Forschungsgemeinschaft, Nederlandse Organisatie voor Wetenschappelijk Onderzoek

OpenAlex ID: W2995869484

FWCI: 0.72

Cites: 20

Cited by: 10

Related to: 10