Year: 2023
Type: preprint
Abstract: This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.
Source: arXiv (Cornell University)
DOI: 10.15480/882.16720
Authors: Ay, Nihat, Oostrum, Jesse van, Datar, Adwait
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Deutsche Forschungsgemeinschaft
Award: G4341881539
OpenAlex ID: W7129368331