Does Adaptive Micro-Batch Triggering Help Spark Structured Streaming? A Fault-Injection Study of Replay, Recovery and Latency
Year: 2026
Type: preprint
Abstract: Checkpoint-interval tuning is usually framed as a trade-off between overhead in normal operation and work lost on failure, and adaptive controllers aim to checkpoint more often when failures look likely. Spark Structured Streaming has no separate checkpoint interval: its processing-time trigger schedules micro-batches, and the offset and commit logs that recovery relies on are written once per batch. We therefore studied adaptive trigger control and asked whether the classic framing carries over. We built a fault-injection harness that SIGKILLs a Spark job reading sequence-numbered records from Kafka, restarts it from its checkpoint, and counts every duplicated or missing record, and implemented a controller (ACS) that changes the trigger at runtime by restarting the query between batches. Over 210 measured trials on one machine with a light, low-rate workload, a crash caused a batch to be reprocessed in 9 of 120 comparison trials, because records that arrive between batches are read for the first time after a restart, not replayed; when it did happen, the replay was the interrupted batch and reached 15,750 records. Recovery took about 10 s, with no systematic relationship to the interval, and was dominated by restart. In this setting the interval traded end-to-end latency against batch-commit count rather than against lost work. Interval switches were cheap (median 0.54 s), and no duplicates were observed in the 320 tested switches. On pairs with matched workload seeds and kill delays, a fixed interval set to ACS's own mean matched or beat it: against a fixed 5.8 s interval, the original ACS rule had the lower 95th-percentile latency in 0/10 pairs and fewer commits in 2/10. Failure-risk-driven adaptation thus showed no measurable benefit under the tested workload; we do not claim the same for heavily loaded or stateful jobs. We also report two measurement pitfalls: an earlier simulation that predicted the wrong failure behaviour, and laptop standby that silently paused six trials, which we detected from the producer's clock.
Author: Medipally, Rajshekar
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222901013