Understanding Spark System Performance for Image Processing in a Heterogeneous Commodity Cluster
Year: 2018
Type: dissertation
Abstract: In recent years, Apache Spark has seen a widespread adoption in industries and institutions due to its\ncache mechanism for faster Big Data analytics. However, the speed advantage Spark provides, especially in\na heterogeneous cluster environment, is not obtainable out-of-the-box; it requires the right combination of\nconfiguration parameters from the myriads of parameters provided by Spark developers. Recognizing this\nchallenge, this thesis undertakes a study to provide insight on Spark performance particularly, regarding\nthe impact of choice parameter settings. These are parameters that are critical to fast job completion and\neffective utilization of resources.\nTo this end, the study focuses on two specific example applications namely, flowerCounter and imageClustering,\nfor processing still image datasets of Canola plants collected during the Summer of 2016 from selected\nplot fields using timelapse cameras in a heterogeneous Spark-clustered environments. These applications\nwere of initial interest to the Plant Phenotyping and Imaging Research Centre (P2IRC) at the University of\nSaskatchewan. The P2IRC is responsible for developing systems that will aid fast analysis of large-scale seed\nbreeding to ensure global food security. The flowerCounter application estimates the count of flowers from\nthe images while the imageClustering application clusters images based on physical plant attributes. Two\nclusters are used for the experiments: a 12-node and 3-node cluster (including a master node), with Hadoop\nDistributed File System (HDFS) as the storage medium for the image datasets.\nExperiments with the two case study applications demonstrate that increasing the number of tasks does\nnot always speed-up job processing due to increased communication overheads. Findings from other experiments\nshow that numerous tasks with one core per executor and small allocated memory limits parallelism\nwithin an executor and result in inefficient use of cluster resources. Executors with large CPU and memory,\non the other hand, do not speed-up analytics due to processing delays and threads concurrency. Further\nexperimental results indicate that application processing time depends on input data storage in conjunction\nwith locality levels and executor run time is largely dominated by the disk I/O time especially, the read\ntime cost. With respect to horizontal node scaling, Spark scales with increasing homogeneous computing\nnodes but the speed-up degrades with heterogeneous nodes. Finally, this study shows that the effectiveness\nof speculative tasks execution in mitigating the impact of slow nodes varies for the applications.
Author: Owolabi O Adekoya
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2941859937
