Work

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression For Distributed Optimization With Convergence Guarantees

Year: 2026

Type: conference-paper

Abstract: Low-rank gradient compression methods, such as PowerSGD, have gained attention in communication-efficient distributed optimization. However, the convergence guarantees of PowerSGD remain unclear, particularly in stochastic settings. In this paper, we show that PowerSGD does not always converge to the optimal solution and provide a clear counter-example to support this finding. To address this, we introduce PowerSGD+, which periodically updates the projection subspace via singular value decomposition, ensuring that it remains aligned with the optimal subspace. We prove that PowerSGD+ converges under standard assumptions and validate its effectiveness through empirical evaluation on large language model tasks.

Source: IEEE International Conference on Acoustics Speech and Signal Processing

DOI: 10.1109/icassp55912.2026.11465020

Authors: Shengping Xie, Chuyan Chen, Kun Yuan

Institution: Peking University

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Natural Science Foundation of China, National Key Research and Development Program of China

OpenAlex ID: W4415088063

FWCI:

Cites:

Cited by:

Related to: