Work

Prediction and Interpretability of HPC I/O Resources Usage with Machine Learning

Year: 2024
Type: preprint
Abstract: I/O management tools proposed for high-performance computing (HPC) environments usually rely on accurate I/O bandwidth estimations to make their decisions. However, finding the correct I/O subsystem configuration that provides this maximal I/O bandwidth is particularly hard. In this work, we focus on finding a good estimate of the number of I/O resources (e.g., OSTs and I/O nodes) that provides the maximal bandwidth while minimizing the system occupation and taking into account the natural I/O variability. We use machine learning techniques to do so, focusing on intrinsic application features and system configurations. We show I/O resource usage is predictable and further study the impact of different features. We also validate our models with four I/O kernels from real applications. Finally, we show that our model, when used for resource allocation, can improve application performance.
Source: HAL
Language: English
Open Access status: green
OpenAlex ID: W4402659261
Cites:
Cited by:
Related to: 10