Year: 2025
Type: dissertation
Abstract: Continuous events streamed from enduring emitters like sensors and IoT devices need to be processed as soon as they arrive to provide near real-time insight when dealing with stream processing.Failing to process data promptly can lead to the loss of valuable information.Considering stream events can grow massively, the current mainstream production-scale stream processing engines (SPEs), such as Apache Flink, Apache Storm, and Spark Streaming, enable scale-out deployment in cluster computers to accelerate processing by involving more workers.Generally, distributed stream processing systems (DSPSs) are able to process unbounded streams of events for a long time period.The input streams may fluctuate in many ways, such as volumes, input rates, and required processing complexity.A robust DSPS must adapt to such potentially unpredictable workloads.When systems need more computing resources, scalingout is performed by including more computing nodes.Conversely, some nodes should be released if the workload decreases, i.e., scaling-in, to utilize resources effectively.In case of failures, systems must be recovered and restored properly to ensure that the processing resumes correctly.Scaling and failure recovery should be performed efficiently to maintain low-latency processing requirements, which is the essential property of stream processing.A common challenge in scaling and recovering from failures in DSPSs is the restoration of states for reconfigured or recovered operators to guarantee valid processing outcomes.The mechanisms must determine the correct properties of the reinstated operators and ensure they reproduce the vanished outputs in a consistent way.Furthermore, each stream event must be identified accurately to achieve exactly-once semantics, the firm guarantee of correctness and completeness in stream processing, where each event contributes once to the processing results, even in the presence of failures.Common methods apply global recovery for state restoration to guarantee such semantics.The global recovery is basically a redeployment of a streaming job, followed by state restoration from a consistent global state, namely a global snapshot.However, the global recovery needs to reinitiate all operators to a checkpoint to maintain state consistency, which leads to processing delays.The availability of checkpoints depends on the frequency of taking global snapshots.Unfortunately, taking a global snapshot requires synchronization among all operators, which can lead to end-to-end performance penalties.Typically, scaling and recovery involve only a particular subset of operators.This thesis focuses on this point and proposes new approaches using partial snapshots for both recovery and scaling.As for recovery, we propose local recovery schemes that localize recovery into a subset of operator instances.The subset is derived from a used-defined snapshot region.To improve the availability of checkpoints, we introduce a partial snapshot, which is taken from a snapshot region.The process of recording a partial snapshot is more efficient than that of a global snapshot, as it has fewer dependencies.The local recovery schemes can leverage partial snapshots for state restoration.Additionally, we also propose a dynamic scaling strategy that prevents unnecessary redeployment of operators.It enables scaling per-operator-basis at runtime, without redeploying streaming jobs.Dynamic scaling is the tailored version of the partial snapshot and local recovery schemes that targets a specific operator and integrates the schemes into a single workflow.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7171273467