Work

Simulation of Large-Scale HPC Storage Systems: Challenges and Methodologies

Year: 2024

Type: conference-paper

Abstract: As the scale of production HPC platforms increases, so does the computing and I/O performance gap, exacerbating the storage bottleneck. High-performance storage systems have been developed to alleviate this bottleneck, but many questions remain concerning their architecture, implementation, and configuration. Answering these questions via experimental campaigns proves arduous. First, some answers are required before deploying the system. Second, once a system hits production the experimental scope is limited by the system's specific configuration and by constraints of production use. In this work we identify challenges posed by the design and validation of a storage simulator. We then propose solutions implemented in Fives, a simulator of HPC workloads on platforms that comprise a parallel file system. We show how our simulator can be instantiated and calibrated for the accurate simulation of a production Lustre deployment.

Source: IEEE International Conference on High Performance Computing, Data, and Analytics

DOI: 10.1109/hipc62374.2024.00031

Authors: Julien Monniot, François Tessier, Henri Casanova, Gabriel Antoniu