Shifted CholeskyQR in FP32: measured breakdown, an a-posteriori bound, and an FP64 orthogonality check in cholqr
Year: 2026
Type: article
Abstract: CholeskyQR-type algorithms compute a thin QR factorization from a few Gram products and small Cholesky factorizations. They are fast on GPUs, but in FP32 they can fail abruptly. Shifted CholeskyQR3 (Fukaya et al., 2020) is the variant implemented in the cholqr library (Fukuda, 2026). Its a-priori guarantee requires conditions that do not hold in FP32 at sizes such as $4096 \times 276$, and its provable shift is much larger than the small shift cholqr uses. This note gives no a-priori breakdown condition for that setting: a sharp condition that holds on all hardware is out of reach, because the FP32 Gram rounding depends on undocumented summation orders and tensor-core behaviour. Instead it collects what can be proved after the fact, and what was measured. - Proved. (i) The per-pass orthogonality bound of Fukaya et al. (2020, eq. (3.33)), restated with all rounding collected in two quantities and used a posteriori, together with two elementary observations on the shifted first pass and on positive definiteness. These exact-arithmetic statements, and a contraction bound for the next pass, are formalized in Lean 4; the floating-point facts they use (rounding of the Gram product, the Cholesky factor, the normalisation) enter as hypotheses and are proved on paper or measured. (ii) An FP64 a-posteriori orthogonality check (verify=True): one FP64 Gram product of the returned $Q$ gives a bound on $\|Q^T Q - I\|_F$ for the exact values of the returned tensor. The bound is rigorous if the assumptions (H0)–(H2) of Sec. 4 hold; (H2), on the FP64 matrix product actually executed, is assumed, not checked. The bound itself is standard; what is new here is engineering (Sec. 4, 11). - Measured (pre-registered). Where FP32 scqr3 breaks, on Apple M4 CPU and GPU, x86 CPUs, NVIDIA GPUs from Turing to Blackwell, one AMD GPU (Instinct MI350X), and a Google TPU v5e (matrix products only; the Cholesky ran on the host CPU). The first breakdown ranged from about 3e5 in condition number (two-cluster and staircase spectra) to 1e8 (e.g. power-law spectrum on the M4 CPU); linear and spike spectra broke anywhere between 1e6 and 1e10, or not at all up to 1e10. The same input can break on one CPU and not on another. With measured rounding terms and no added margin, the bound held in every checked FP32 pass. The orthogonality check had no false acceptance on CPU, Apple GPU, an RTX 4090 and an AMD MI350X (bound computed on the host CPU). On the RTX 4090, 230 unrepaired outputs had a true orthogonality error above 1e-4 (precision settings "high" and "highest" together; 96 of them from uncompensated TF32, the rest from breakdowns, low-precision outputs and a rank-deficient batch); the empirical detector flagged 164 of them (38 of the 96 TF32 cases), and the check rejected all 230. - Empirical, not proved. The run-time detector's threshold and the rule for when an FP32 Cholesky fails. Both are validated only on data not used to choose them; the threshold only for $l \le 276$ (at $l = 512$ and 1024 it missed a few marginally degraded runs, Sec. 7). A size-dependent threshold (not in 0.5.1; planned for the next release) passed a pre-registered CPU test at $l = 384$, 768 and 1536 ($m = 8l$; Sec. 7). We also report TF32 and a 3xTF32 compensation (better, not full FP32, and slower), and list every registered prediction that failed, with a diagnosis, and the protocol deviations (Sec. 8, Appendix A).
Author: Fukuda, Soki
Institution: The University of Tokyo
Language: English
Topic: Matrix Theory and Algorithms
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225996018
