Statistical Guarantees for Generative Models as Distribution Estimators
Year: 2025
Type: book
Abstract: Generative models have emerged as a symbol of artificial intelligence, enabling computers to mimic human behavior based on large datasets. While the empirical results are impressive, the theoretical understanding lags behind. Naturally, using more training data should result in a better model. In a rigorous mathematical setting, we aim to bound a model's error by a declining function of the number of samples. In this thesis, we study such upper bounds for two models: Generative Adversarial Networks (GANs) and Flow Matching. Furthermore, we will extend Flow Matching to the setting of conditional distribution estimation. Along the way, we will also investigate classical kernel-based methods for distribution estimation. Since their introduction in 2014, GANs have evolved from the initial Vanilla setup to several adaptations. The statistical literature mainly focuses on Wasserstein GANs and their generalizations, which can build on the theory of optimal transport. In contrast, statistical results for Vanilla GANs are limited to very specific settings. To bridge this gap, we establish a connection between Vanilla GANs and the Wasserstein-$1$ distance by leveraging the neural network architecture commonly used in practice. This enables us to transfer various results, such as dimension reduction properties, to Vanilla GANs. Our findings finally explain the empirical success of these early generative models. Flow Matching is a very recent generative model introduced in 2023 that just emerged as an alternative to diffusions, the current state-of-the-art. Consequently, theoretical results are limited. First, we demonstrate the natural connection between Flow Matching and the kernel density estimator. Then, we prove that, in an over-parameterized setting, Flow Matching achieves minimax optimal rates in the Wasserstein distance. Additionally, we study the regularity of the underlying dynamics, which are essential to statistical bounds. This enables us to derive rates using smaller networks, improving and extending the few preceding results. Subsequently, we study Flow Matching in a conditional setting. This allows us to employ Flow Matching in a forecasting context. After extending the model in a mathematically reasonable way and showing the connection to a Nadaraya-Watson-type estimator, we connect proper scoring rules, which are a common way to measure model error in forecasting settings, to the concept of risk in statistical learning. Then we study rates of convergence in the risk associated to the Fourier-Score. After deriving lower bounds, we prove that the Nadaraya-Watson-type estimator is minimax optimal. For certain dimensions, we extend our results to the conditional Flow Matching estimator. Finally, we demonstrate Flow Matching's practical capability to estimate a conditional distribution.
Source: KITopen
Author: Kunkel, Lea Maria
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7116066730
FWCI:
Cites:
Cited by:
Related to: