The Earth Moves, But So Does the Bias: Systematic Upward Bias of the Wasserstein (Earth Mover's) Distance and Permutation-Based Null Calibration
Year: 2026
Type: preprint
Abstract: The Earth Mover's Distance (EMD) is gaining increasing interest among political scientists for assessing similarity in preference distributions. Existing applications predominantly involve relatively large samples and low-dimensional response spaces, which can mitigate finite-sample distortions. However, empirical EMD remains biased upward due to sampling variation in empirical probability measures. This problem is often overlooked and becomes especially severe in high-dimensional or sparse settings, including conjoint distributions that serve as an illustrative example in this paper. As political scientists are broadening their use cases of EMD, this paper cautions against interpreting standard bootstrap uncertainty bounds as a correction for the upward bias of empirical EMD. It proposes a permutation-based null calibration framework for more robust hypothesis testing. As a non-parametric approach, it frees researchers from making directional or distributional shape assumptions. While alternative estimators require these rigid assumptions to correct for upward bias, political science data often fail to meet them in practice. Through four sets of Monte Carlo simulations, this paper demonstrates the utility of this framework. The proposed approach also applies more generally to empirical comparisons of two probability distributions defined on a common metric space, provided that the ground distance between support points is substantively meaningful.
Source: arXiv (Cornell University)
Author: Hung, Ho Ting
Institution: University of Oxford
Language: English
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7202175912