Year: 2021
Type: preprint
Abstract: Finding the optimal hyperparameters of a model can be cast as a bilevel optimization problem, typically solved using zero-order techniques. In this work we study first-order methods when the inner optimization problem is convex but non-smooth. We show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian. Using implicit differentiation, we show it is possible to leverage the non-smoothness of the inner problem to speed up the computation. Finally, we provide a bound on the error made on the hypergradient when the inner optimization problem is solved approximately. Results on regression and classification problems reveal computational benefits for hyperparameter optimization, especially when multiple hyperparameters are required.
Source: arXiv (Cornell University)
Authors: Bertrand, Quentin, Klopfenstein, Quentin, Massias, Mathurin, Blondel, Mathieu, Vaiter, Samuel, Gramfort, Alexandre, Salmon, Joseph
Institutions: Institut national de recherche en sciences et technologies du numérique, Commissariat à l'Énergie Atomique et aux Énergies Alternatives, Université Paris-Saclay, Université de Bourgogne, Institut de Mathématiques de Bourgogne, Google (United States), University of Genoa, Institut Montpelliérain Alexander Grothendieck
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Agence Nationale de la Recherche
OpenAlex ID: W3157908369