Work

Distributed Data Processing Frameworks for Real-Time Analytics Systems

Year: 2026
Type: conference-paper
Abstract: With the rapid proliferation of Internet of Things (IoT), cloud computing, edge computing, social media and industrial cyber-physical systems, high velocity data streams are becoming increasingly common that need efficient real-time processing and analytics. Traditional batch-oriented data processing paradigms are frequently insufficient to scale to the volume, velocity and variety of modern streaming data sources, while keeping latency low and being scalable. This paper presents a Distributed Data Processing Framework (DDPF) for Real-Time Analytics Systems that consists of Distributed Stream Processing, Intelligent Resource Scheduling, Adaptive Load Balancing, Fault Tolerance, and Parallel Data Analytics. The proposed framework is based on a hybrid edge-cloud system, which processes the streaming data closer to the source but also leverages the cloud resources for large scale analytical tasks. Optimisation algorithms, based on machine learning techniques, are integrated to allocate the computational resources dynamically, anticipate the workload fluctuations and reduce processing delays. Moreover, it also provides automatic fault recovery and distributed storage management to guarantee service continuity and high availability. The different performance criteria which are adopted to test the performance of the proposed framework are the throughput, latency, scalability, resource utilization, fault recovery time and processing efficiency which are standard real-time analytics metrics. Experimental analysis shows that the proposed framework significantly outperforms the traditional distributed processing system in terms of processing throughput, end-to-end latency, processing scalability and resource utilization. The proposed solution provides an efficient, scalable, and intelligent platform that can be used in various applications, such as smart cities, industrial automation, healthcare monitoring, financial analytics, intelligent transportation systems, and large-scale IoT deployments, where real-time decision-making is required.
Language: English
Open Access status: hybrid
OpenAlex ID: W7220844979
FWCI:
Cites: 15
Cited by:
Related to: 10