Year: 2025
Type: article
Abstract: Traditional centralized systems cannot deal with the big data context. Distributed computing platforms such as Apache Spark have been widely adopted, but configuring their parameters is challenging given the number of factors and their interactions. This work employs Design of Experiments (DoE) techniques to screening most relevant factors regarding execution time of a Naïve Bayes machine learning distributed task on a subset of the PT7 Web Corpus, which has 14.88 GB of data. Employing a fractional factorial design with 192 experimental units and linear regression techniques with backward elimination, we obtained (i) the most relevant factors based on statistical significance and (ii) a model capable of predicting execution time according to parameters' values in the analyzed context. Our results also include a visualization technique based on Inselberg's Parallel Coordinates to comprehend the impact on performance facing various configuration possibilities.
Authors: Maria C. L. Nunes, Jairson B. Rodrigues
Institution: Universidade Federal do Vale do São Francisco
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,490
OpenAlex ID: W4410850695