Robust and reproducible evaluation framework for population synthesis models—Application to probabilistic and deep generative models
Year: 2026
Type: article
Abstract: Population synthesis benefits from a large literature on model implementation; however, there remains no consensus on how to evaluate a generated synthetic population. The generated population must be similar to the real population. That involves ensuring a similar distribution at a macroscopic level, and that each generated individual is realistic at the microscopic level. Given that population synthesis involves real data, another concern is to respect the data privacy, which must be also evaluated. In this paper, leveraging insights from population synthesis and tabular data generation literature, we propose three metrics, that address the current limitations in evaluation. To support our metrics, we provide some mathematical demonstrations, and the robustness of the metrics are tested, whenever possible. In addition to these metrics, we propose an open-source dataset containing the full population, which can serve as reference for model evaluation. This allows proposing a complete evaluation framework for population synthesis promoting reproducible science. As an application, we present an extensive benchmark with probabilistic models, and deep generative models, including –the current state-of-the-art generative model for tabular data synthesis– diffusion model. In this benchmark, we simulate two data scenarios corresponding to real-world use cases. The results indicate that, under these settings, Bayesian networks are the most interesting models, performing well for all criteria. Regarding deep generative models, the diffusion model gives promising results, as it is the only model achieving similar results to those of the Bayesian Network, and could be a great option in more complicated use cases, with more attributes. Our code is available here for future research: https://github.com/vdarsel/PopulationSynthesis.
Authors: Vianey Darsel, Etienne Côme, Latifa Oukhellou
Institution: Université Gustave Eiffel
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $4,040
OpenAlex ID: W7135240522