Performant Unified GPU Kernels for Portable Singular Value Computation Across Hardware and Precision
Year: 2025
Type: conference-paper
Abstract: This paper presents a portable, GPU-accelerated implementation of a QR-based singular value computation algorithm in Julia. The singular value decomposition (SVD) is a fundamental numerical tool in scientific computing and machine learning, providing optimal low-rank matrix approximations. Its importance has increased even more in large-scale machine learning pipelines, including large language models (LLMs), where it enables low-rank adaptation (LoRA). The implemented algorithm is based on the classic two-stage QR reduction, consisting of successive matrix reduction to band form and bidiagonal form. Our implementation leverages Julia’s multiple dispatch and metaprogramming capabilities, integrating with the GPUArrays and KernelAbstractions frameworks to provide a unified type and hardware-agnostic function. It supports diverse GPU architectures and data types, and is, to our knowledge, the first GPU-accelerated singular value implementation to support Apple Metal GPUs and half precision. Performance results on multiple GPU backends and data types demonstrate that portability does not require sacrificing performance: the unified function outperforms most linear algebra libraries (MAGMA, SLATE, rocSOLVER, oneMKL) for matrix sizes larger than 1024 × 1024, and achieves 80%-90% of the performance of cuSOLVER for large matrices.
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: SWQU: Composable Next Generation Software Framework for Space Weather Data Assimilation and Uncertainty Quantification, eMB: Collaborative Research: Discovery and calibration of stochastic chemical reaction network models, Collaborative Research: Frameworks: Convergence of Bayesian inverse methods and scientific machine learning in Earth system models through universal differentiable programming, Collaborative Research: Research Infrastructure: CIRC: ENS: Bridging Science and AI Through Differentiable Program Transformations, G8163885039, G8411644938, WE PROPOSE THE CREATION OF A SINGLE-DISCIPLINE CENTER (SDC) FOR THE EXASCALE SIMULATION OF MATERIALS IN
HYPERSONIC FLOW ENVIRONMENTS. OUR OBJECTIVES ARE AS FOLLOWS. FIRST, WE TARGET A SIGNIFICANT ADVANCE IN THE
ABILITY TO PREDICT THE DEGRADATION (OXIDATION AND MELTING) OF COMPLEX (DISORDERED AND MULTI-COMPONENT) MATERIALS
UNDER EXTREME LOADING, INACCESSIBLE TO DIRECT EXPERIMENTAL OBSERVATION. THIS APPLICATION REPRESENTS A
TECHNOLOGY DOMAIN OF INTENSE CURRENT INTEREST, AND HAS CLEAR EXTENSIONS TO A MUCH WIDER CLASS OF PROBLEMS—
I.E., THOSE INVOLVING MATERIAL INTERFACES IN EXTREME ENVIRONMENTS. INDEED, INTERFACES ARE CRUCIAL TO OUR ABILITY
TO UNDERSTAND, ENGINEER, AND CONTROL MATERIAL BEHAVIOR. YET THE PREDICTIVE SIMULATION OF INTERFACE PROPERTIES
AND THEIR IMPACT REMAINS EXTREMELY CHALLENGING. CAPTURING THE IMPACT OF INTERFACES REQUIRES RESOLVING INTERFACIAL
ATOMIC STRUCTURE AND THUS APPEALING TO FIRST PRINCIPLES (I.E., QUANTUM MECHANICAL) SIMULATIONS; AT THE
SAME TIME, INTERFACIAL STRUCTURES ARE AFFECTED BY CONDITIONS (E.G., THERMAL, MECHANICAL, AND CHEMICAL LOADING)
AT THE ENGINEERING SCALE. THIS PROBLEM THUS PRESENTS AN ENORMOUS AND FUNDAMENTAL MULTISCALE SIMULATION
CHALLENGE, WITH MYRIAD APPLICATIONS—TO HYPERSONICS AND FAR BEYOND., Collaborative Research: CAIG: Interpretable, Stable, Mass-Conserving AI for Air Pollution Modeling
OpenAlex ID: W4415195033