Year: 2024
Type: article
Abstract: Java-based big data frameworks such as Apache Hadoop, Spark, and Flink play a critical role in large-scale data processing within distributed environments. Despite their widespread adoption, these frameworks encounter several technical and operational challenges that can affect performance, scalability, and reliability. The purpose of this study is to examine the key challenges faced by Java-based big data frameworks when deployed in distributed systems. The research adopts a qualitative methodology based on an extensive review of existing literature, technical documentation, and case studies related to distributed computing and big data processing. The findings reveal that major challenges include JVM overhead, garbage collection latency, memory management complexities, network and I/O bottlenecks, fault tolerance overhead, and difficulties in deployment, configuration, and security management. These issues can lead to reduced efficiency and increased operational costs in large-scale environments. The study concludes that while Java-based big data frameworks remain robust and versatile, addressing these challenges through optimized memory handling, improved resource management, and cloud-native architectural enhancements is essential for achieving better performance and reliability in distributed environments.
Author: Yusuf Adebayo
Institution: Ladoke Akintola University of Technology
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7128426077