Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations
Year: 2022
Type: conference-paper
Abstract: Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabytescale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data “hogs” disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the nextsteps towards its use in production.
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Partnership to Advance Throughput Computing (PATh), U.S. CMS Operations at the LHC, S2I2: Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP), THE OPEN SCIENCE GRID The Next Five Years: Distributed High Throughput Computing for the Nation's Scientists, Researchers, Educators, and Students, Collaborative Research: Data Infrastructure for Open Science in Support of LIGO and IceCube
OpenAlex ID: W4317826930