Work

Efficient SGD Neural Network Training via Sublinear Activated Neuron Identification

Year: 2024

Type: conference-paper

Abstract: Deep learning has been widely used in many fields, but the model training process usually consumes massive computational resources and time. Therefore, designing an efficient neural network training method with a provable convergence guarantee is a fundamental and important research question. In this paper, we present a static half-space report data structure that consists of a fully connected two-layer neural network for shifted ReLU activation to enable activated neuron identification in sublinear time via geometric search. We also prove that our algorithm can converge in O(M2/ϵ2) time with network size quadratic in the coefficient norm upper bound M and error term ϵ.

Source: IEEE International Conference on Big Data (BigData)

DOI: 10.1109/bigdata62323.2024.10825793

Authors: Lianke Qin, Zhao Song, Yuanyuan Yang

Institutions: University of California, Santa Barbara, University of California, Berkeley, University of Washington

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4406458965

FWCI: 1.178

Cites: 44

Cited by: 4

Related to: 10