GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading
Year: 2024
Type: article
Abstract: Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first Exascale computer, named Frontier, comprising AMD MI250X GPUs with HPE's Slingshot interconnect, operated by the U.S Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCM library calls. Data movement between between the CPU host and the GPU device (which can be inefficient) is avoided almost entirely except when needed for some I/O operations, while the memory footprint including the size of communication buffers is managed carefully in order to accommodate the largest problem sizes possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from $2048^3$ (which fits on a single node) to $32768^3$ (which requires at least 4096 Frontier nodes or more of 9408 on the system). Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are considered. The present code suite (identified by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (measured in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance data indicate very good results in both weak scaling and strong scaling, with notable departures only for $2048^3$ where communication traffic may be entirely intra-node, and $32768^3$ where a challenge due to small message sizes does arise. Communication performance is addressed further using a small kernel that performs all-to-all communication calls in configurations matching those in the full-featured turbulence simulation code. Overall, although not perfect, the scalability achieved at the extreme problem size of $32768^3$ (and up to 8192 nodes --- which corresponds to hardware rated at just under 1 Exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled directly by the present work as well as algorithmic extensions covering turbulent mixing and dispersion are expected in the near future.
Source: Computer Physics Communications
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,410
OpenAlex ID: W4402284386