Optimizing Data Stream Processing Pipelines: Using In-Memory DB and Change Data Capture for Low - Latency Enrichment
Year: 2022
Type: article
Abstract: This paper presents a new way of optimization of data stream processing pipelines using Redis, an in - memory data store, and Change Data Capture for real - time data synchronization. We will detail how this combination reduces latency during the data enrichment process-for instance, one critical building block for nearly all stream processing architectures today. Our experiments show massive improvements in the speed and efficiency of processing as against traditional approaches, peaking at 80% latency reduction and 3x increase in throughput. On - demand scalable solutions to large volume and real - time data streams are enabled in application domains such as financial analytics, IoT, and social media analytics.
Author: Purshotam S Yadav
Institution: Georgia Institute of Technology
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4403337517
