An uncertainty-based objective function for hyperparameter optimization in Gaussian processes applied to expensive black-box problems
Year: 2024
Type: article
Abstract: As of today, Gaussian processes (GPs) have been widely and successfully used in the context of design optimization based on expensive-to-evaluate functions. This supervised learning method enables a generation of accurate nonlinear surrogate models based on relatively small datasets. Nonetheless, their most valuable asset is to provide uncertainty in predictions. Despite their excellent stochastic properties, Gaussian processes are unfortunately not immune to threats such as the generation of distorted predictions, especially when the amount of data available is very limited. This shortcoming is caused by a poor choice of the GP hyperparameters and represents a serious threat to the efficiency and effectiveness of the whole surrogate-based optimization. In this paper we present the Hybrid Loss (HL), a novel uncertainty-aware objective function for the hyperparameter tuning of Gaussian processes. This method is intended to exploit information coming from the predictive variance to remedy the typical shortcomings of the log marginal likelihood, i.e. the objective function commonly used to optimize GP hyperparameters. By pairing this methodology with a well-known adaptive sampling strategy, we investigate the performance on a wide range of benchmark functions and a real engineering problem. The observed evidence clearly shows how uncertainty can be successfully exploited to make a wiser choice of the hyperparameters. This translates into more accurate predictions, surrogate models less prone to overfitting, and above all, greatly improved convergence rates.
Source: Applied Soft Computing
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,620
Award: G9000141011
OpenAlex ID: W4391554642