Work

In Silico World - D6.3 Efficiency recovery strategies report

Year: 2024

Type: article

Abstract: The purpose of this deliverable is to report on strategies adopted in the ISW project to ensure operational efficiency and smooth processing of large-scale workflows with the use of research models developed in the project, and through the Model Execution Environment (MEE) (1), identifying and resolving performance bottlenecks which arise when computations are submitted to the available High Performance Computing resources. The cornerstone of the developed solution is rich monitoring of the scientific applications and the HPC computing infrastructure based on the modern observability principles. Observability adopts a holistic approach, using a suite of methodologies, strategies, and tools designed to gather and analyze detailed telemetry data. The primary goal of observability is to uncover unrecognized issues and investigate the ``unknown unknowns'' that plague the underlying computer systems and applications (2).Existing observability technologies emerged in the context of cloud-native systems and commercial applications, hence their adoption to scientific applications in HPC environments is not straightforward. Key challenges include collection of application-level metrics, instrumentation of scientific applications to collect rich telemetry data, and tracing of distributed call chains in the HPC execution ecosystem. The solution described in this document is based on open-source standard and tools: the OpenTelemetry (3) observability framework, the OpenSearch (4) data search and analysis engine, Grafana (5), and JupyterHub (6) providing a notebook-based environment for advanced analysis of the collected telemetry data. The remainder of this document is structured as follows. Section 2 briefly describes the scientific applications monitored using the proposed system. Section 3 introduces the architecture and implementation details of the monitoring system. Section 4 presents examples of performance monitoring and analytical capabilities of developed system. Section 5 highlights optimization conducted based on the data analytics capabilities of the monitoring system. Section 6 concludes work.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.14843507

Authors: Balis, Bartosz, Czerepak, Konrad, Zając, Karol, Meizner, Jan, Zhyhulin, Taras

Institutions: AGH University of Krakow, Akademickie Centrum Komputerowe Cyfronet AGH, Sano – Centre for Computational Personalised Medicine International Research Foundation

Language: English

Topic: Software System Performance and Reliability

Subfield: Computer Networks and Communications

Field: Computer Science

Domain: Physical Sciences

SDG: Industry, innovation and infrastructure

Open Access status: green

Funder: European Commission