Year: 2025
Type: conference-paper
Abstract: OverviewLarge scientific instruments are shared resources that are often coupled with advanced computing resources, such as HPC clusters, powerful storage systems, and, increasingly, machine learningoriented hardware.Each of these resources may be a multi-user system shared among disparate teams.In the context of real-time data acquisition and analysis, sharing can have a negative impact on the ability of the computing system to satisfy scientific objectives.In this study, we focus on the impact of a shared filesystem (FS), such as NFS or GPFS, on the scientific workflows that move and analysis variable-sized data sets.We then present a preliminary study on the use of predictive methods to capture and potentially feed back information about filesystem usage to the scientific goal level. MotivationThe overarching goal of the work presented here is to support automated, near-real-time scientific data acquisition and analysis by enhancing reliability through performance prediction and anomaly detection.This will be delivered as a service to workflow-level services that are capable of responding to dynamic resource availabilities and capabilities, automatically steering data collection in support of scientific goals.Communication within this workflow, including among monitoring and predictive components, will be performed over a reliable Kafka-like service [2,3].The first step in such an effort is to select or develop a predictive model that can make reasonable predictions.To do this, we need to collect representative data.In this paper, we present a representative
Authors: Justin Wozniak, Sushil Regmi, Tong Shu, Ian Foster
Institutions: Argonne National Laboratory, University of North Texas
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G102163711, G2287611682, G262962599, CAREER: A Time-Sensitive Large Model Training Platform for Dynamic Data Analytics, G498139845, G513797252, G5398989114, G6848031779, G6918803902, Collaborative Research: CyberTraining: Pilot: Research Workforce Development for Deep Learning Systems in Advanced GPU Cyberinfrastructure, G878601127, G969889393, G993149817
OpenAlex ID: W4411663232