Year: 2024
Type: article
Abstract: Java-based big data frameworks have become foundational technologies for large-scale data storage, processing, and analytics. This study examines the architectural patterns that underpin widely adopted frameworks such as Apache Hadoop, Spark, Flink, and Kafka, with the purpose of identifying how these patterns address scalability, fault tolerance, performance, and maintainability challenges in distributed environments. The research employs a qualitative, comparative analysis of framework architectures, official documentation, and existing literature to identify recurring design patterns and their practical implementations. The key findings reveal that patterns such as layered architecture, master–worker coordination, dataflow pipelines, event-driven processing, and shared-nothing design are consistently applied across frameworks, albeit with variations tailored to batch or stream processing requirements. Additionally, fault-tolerance mechanisms like replication, checkpointing, and lineage-based recovery are shown to be critical in achieving reliability at scale within the constraints of the Java Virtual Machine. The study concludes that the effective combination of these architectural patterns is central to the success of Java-based big data frameworks, and that understanding these patterns can guide architects and developers in selecting, designing, and optimizing distributed data processing systems.
Author: Toheeb Mudasir
Institution: Obafemi Awolowo University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7128444179