Work

Worth Their Weight: Randomized and Regularized Block Kaczmarz Algorithms without Preprocessing

Year: 2026

Type: article

Abstract: . Due to the ever growing amounts of data leveraged for machine learning and scientific computing, it is increasingly important to develop algorithms that sample only a small portion of the data at a time. In the case of linear least-squares, the randomized block Kaczmarz method (RBK) is an appealing example of such an algorithm, but its convergence is only understood under sampling distributions that require potentially prohibitively expensive preprocessing steps. To address this limitation, we analyze RBK when the data is sampled uniformly, showing that its iterates converge in a Monte Carlo sense to a weighted least-squares solution. Unfortunately, for general problems the bias of the weighted least-squares solution and the variance of the iterates can become arbitrarily large. We show that these quantities can be rigorously controlled by incorporating regularization into the RBK iterations, yielding the regularized algorithm ReBlocK. Numerical experiments including examples arising from natural gradient optimization demonstrate that ReBlocK can outperform both RBK and minibatch stochastic gradient descent for inconsistent problems with rapidly decaying singular values. Reproducibility of computational results. This paper has been awarded the “SIAM Reproducibility Badge: Code and data available” as a recognition that the authors have followed reproducibility principles valued by SIMAX and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/ggoldsh/block-kaczmarz-without-preprocessing and in the supplementary materials ( experiments.zip [1.05MB]). [Formula: see text]

Source: SIAM Journal on Matrix Analysis and Applications

DOI: 10.1137/25m1764372

Authors: Gil Goldshlager, Jiang Hu, Lin Lin

Institutions: University of California, Berkeley, Tsinghua University, Lawrence Berkeley National Laboratory

Language: English

Topic: Matrix Theory and Algorithms

Subfield: Computational Theory and Mathematics

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: U.S. Department of Energy, Simons Foundation, Office of Science, Office of Naval Research

Awards: G3277760478, 2022 COMPUTATIONAL SCIENCE GRADUATE FELLOWSHIP PROGRAM, 2022 COMPUTATIONAL SCIENCE GRADUATE FELLOWSHIP PROGRAM, G8876996369, G9001168218