Year: 2025
Type: preprint
Abstract: When training large models on limited data, avoiding overfitting is paramount. Common grid search or smarter search methods rely on expensive separate runs for each candidate hyperparameter, while carving out a validation set that reduces available training data. In this paper, we study gradient-based learning of hyperparameters via the evidence lower bound (ELBO) objective from Bayesian variational methods. This avoids the need for any validation set. We focus on scenarios where the model is over-parameterized for flexibility and the approximate posterior is chosen to be Gaussian with isotropic covariance for tractability, even though it cannot match the true posterior. In such scenarios, we find the ELBO prioritizes posteriors that match the prior, leading to severe underfitting. Instead, we recommend a data-emphasized ELBO that upweights the likelihood but not the prior. In Bayesian transfer learning of image and text classifiers, our method reduces the 88+ hour grid search of past work to under 3 hours while delivering comparable accuracy. We further demonstrate how our approach enables efficient yet accurate approximations of Gaussian processes with learnable lengthscale kernels.
Source: arXiv (Cornell University)
Authors: Harvey, Ethan, Petrov, Mikhail, Hughes, Michael C.
Institution: Tufts University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Decision-Aware Learning of Adaptive Probabilistic Models from Limited Supervision, Covert Cerebrovascular Disease Detected by Artificial Intelligence (C2D2AI): Pragmatic Neuroimaging Biomarkers for Future Stroke and Dementia Risk, CC* Compute: GPU Infrastructure to Explore New Algorithmic & AI Methods in Data-Driven Science and Engineering at Tufts University
OpenAlex ID: W4407185112