Year: 2026
Type: conference-paper
Abstract: Real-time scientific workflows operate on continuous data streams and must produce timely, high-quality results despite executing on complex, failure-prone infrastructure. Hardware faults, network disruptions, and performance anomalies caused by resource contention or system heterogeneity can severely degrade performance and violate real-time constraints. We focus on strengthening the resilience of the producer–consumer streaming pattern, a fundamental building block of scientific streaming workflows. We present two complementary techniques: (i) a dynamic, asynchronous, non-blocking checkpointing mechanism that preserves progress without interrupting computation, and (ii) a progress-aware load redistribution strategy that detects slow workers and proactively rebalances tasks. Together, these mechanisms maintain forward progress and balanced execution even in highly error-prone environments. Experimental results show that our approach reduces the impact of failures and performance anomalies by up to 6 ×, while introducing less than 1% overhead in failure-free execution.
Institutions: Argonne National Laboratory, University of Chicago
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G102163711, G262962599, Collaborative Research: Frameworks: MPI4AI: Enhancing Performance and Productivity of AI Science through Next-generation High Performance Communication Abstractions, Collaborative Research: Elements: VLCC-States: Versioned Lineage-Driven Checkpointing of Composable States, G513797252, G6848031779, G969889393
OpenAlex ID: W7167027240