Information criterion for approximation of unnormalized densities
Year: 2025
Type: article
Abstract: This paper considers the problem of approximating an unknown density when it can be evaluated up to a normalizing constant at a finite number of points. This density approximation problem is ubiquitous in statistics, such as approximating a posterior density for Bayesian inference and estimating an optimal density for importance sampling. We consider a parametric approximation approach and cast it as a model selection problem to find the best model in pre-specified distribution families (e.g., select the best number of Gaussian mixture components and their parameters). This problem cannot be addressed with traditional approaches that maximize the (marginal) likelihood of a model, for example, using the Akaike information criterion (AIC) or Bayesian information criterion (BIC). We instead aim to minimize the cross-entropy that gauges the deviation of a parametric model from the target density. We propose a novel information criterion called the cross-entropy information criterion (CIC) and prove that the CIC is an asymptotically unbiased estimator of the cross-entropy (up to a multiplicative constant) under some regularity conditions. We propose an iterative method to approximate the target density by minimizing the CIC. We demonstrate how the proposed method selects a parametric model that well approximates the target density through multiple numerical studies in the Supporting Information.
Source: PLoS ONE
Authors: John Y Choe, Yen-Chi Chen, Nick Terry
Institution: University of Washington
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,382
Award: G4906632287
OpenAlex ID: W4408520662