Year: 2017
Type: preprint
Abstract: With the ever-growing need of data in HPC applications, the congestion at the I/O level becomes critical in super-computers. Architectural enhancement such as burst-buffers and pre-fetching are added to machines, but are not sufficient to prevent congestion. Recent online I/O scheduling strategies have been put in place, but they add an additional congestion point and overheads in the computation of applications. In this work, we show how to take advantage of the periodic nature of HPC applications in order to develop efficient periodic scheduling strategies for their I/O transfers. Our strategy computes once during the job scheduling phase a pattern where it defines the I/O behavior for each application, after which the applications run independently, transferring their I/O at the specified times. Our strategy limits the amount of I/O congestion at the I/O node level and can be easily integrated into current job schedulers. We validate this model through extensive simulations and experiments by comparing it to state-of-the-art online solutions, showing that not only our scheduler has the advantage of being de-centralized and thus overcoming the overhead of online schedulers, but also that it performs better than these solutions, improving the application dilation up to 13% and the maximum system efficiency up to 18%.
Source: arXiv (Cornell University)
Institutions: Université de Bordeaux, TADAAM: Gestion des données à l'échelle du système en fonction de la topologie pour le calcul haute performance, University of Illinois Urbana-Champaign, Mellanox Technologies (United States), École Normale Supérieure de Lyon, Laboratoire de l'Informatique du Parallélisme, ROMA: Optimisation des ressources : modèles, algorithmes et ordonnancement
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Vanderbilt University
OpenAlex ID: W2954371657