Work

Engineering Scalable Messaging Applications Using Stream Processing, Microservices, and Latency-Aware Data Pipelines

Year: 2020
Type: article
Abstract: Ultra-low latency and high-throughput messaging are demanded by modern real-time apps (financial trading platform, IoT telemetry system, online gaming infrastructures and massive analytics pipelines etc). But the conventional monolithic and batch based messaging architecture is unable to support the strict performance needs when faced with active and concurrent loads. The current distributed streaming solutions are scalable, yet they do not typically offer the built-in latency-aware optimization, dynamic scaling, and unified fault-tolerance, specific to microservices-based environments and established on clouds. This study overcomes these drawbacks by establishing this gap in high-throughput stream processing and architecture design sensitive to latency, especially in systems requiring exactly-once semantics, quick failure recovery and scalability when required to be elastic across a distributed cluster. We suggest a scalable, latency-conscious microservices-based message system to seal this gap incorporating both distributed event streaming provided by Apache Kafka and real time processing of streams provided by Apache Flink as well as orchestration of the new environment provided by Kubernetes. Adaptive partitioning, checkpoint-enabled fault tolerance, horizontal auto scaling, and backpressure-sensitive data flows are built in to the framework. The experimental assessment shows a 40-60 percent drop in end-to-end latency and almost linear increase in throughput due to addition of more workload, quick fault recovery within a few seconds and enhanced energy efficiency in using the CPU, when compared with the conventional batch architecture. These are the contributions such as a latency optimal architectural design, exhaustive scalability/fault tolerance design and validated performance benchmarks that are capable of supporting next generation distributed real-time systems.
Language: English
Open Access status: bronze
OpenAlex ID: W7142291143
FWCI:
Cites: 21
Cited by:
Related to: 10