Work

A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training

Year: 2018

Type: preprint

Abstract: Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optimization task. Recent work have shown the efficiency of block coordinate descent (BCD) type methods empirically for training DNNs. In view of this, we propose a novel algorithm based on the BCD method for training DNNs and provide its global convergence results built upon the powerful framework of the Kurdyka-Lojasiewicz (KL) property. Numerical experiments on standard datasets demonstrate its competitive efficiency against standard optimizers with backprop.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.1803.09082

Authors: Lau, Tim Tsz-Kit, Zeng, Jinshan, Wu, Baoyuan, Yao, Yuan

Institutions: Hong Kong University of Science and Technology, Jiangxi Normal University, Tencent (China), Peking University

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W2795073474

Cites:

Cited by: 18

Related to: 20