Work

Empirical Evaluation of Acquisition Functions for Bayesian Optimization-Based Configuration Tuning of Apache Spark Applications

Year: 2025

Type: article

Abstract: The execution time of an Apache Spark application is heavily influenced by its configuration settings. Accordingly, Bayesian Optimization (BO) is commonly used for automated tuning, employing the acquisition function, Expected Improvement (EI). However, existing works did not compare the performance to the other acquisition functions empirically. In this paper, we show that EI may not work well for Spark applications due to a huge search space compared to the other optimization problems. In addition, we demonstrate the performance of BO based on Probability of Improvement (PI), which achieves exploration via rich random initialization and exploitation via the PI acquisition function. Through the experimental evaluations, we show that the PI-based BO outperforms the EI-based BO in both optimal time and optimization cost.

Source: IEICE Transactions on Information and Systems

DOI: 10.1587/transinf.2024edl8071

Authors: Hyunsik YOON, Yon Dohn CHUNG

Institutions: Korea University, Korea University

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funders: National Research Foundation, National Research Foundation of Korea, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion