Work

Probabilistic Measures Afford Fair Comparisons of AIWP and NWP Model Output

Year: 2025
Type: article
Abstract: We introduce a new measure for fair and meaningful comparisons of single-valued output from artificial-intelligence-based weather prediction (AIWP) and numerical weather prediction (NWP) models, called potential continuous ranked probability score (PCRPS). In a nutshell, we subject the deterministic backbone of physics-based and data-driven models post hoc to the same statistical postprocessing technique, namely, isotonic distributional regression (IDR). Then, we find PCRPS as the mean continuous ranked probability score (CRPS) of the postprocessed probabilistic forecasts. The nonnegative PCRPS measure quantifies potential predictive performance and is invariant under strictly increasing transformations of the model output. PCRPS attains its most desirable value of zero if, and only if, the weather outcome is a fixed, nondecreasing function of the model output. The PCRPS measure is recorded in the unit of the outcome, has an upper bound of one-half times the mean absolute difference between outcomes, and serves as a proxy for the mean CRPS of real-time, operational probabilistic products. When applied to WeatherBench 2 data, our approach demonstrates that the data-driven GraphCast model outperforms the leading physics-based European Centre for Medium-Range Weather Forecasts (ECMWF) high-resolution (HRES) model. Furthermore, the PCRPS measure for the HRES and GraphCast models aligns exceptionally well with the mean CRPS of the operational ECMWF ensemble and the GenCast ensemble, respectively. Across application domains, our approach affords comparisons of single-valued forecasts in settings where the prespecification of a loss function—which is the usual, and principally superior, procedure in forecast contests, administrative, and benchmark settings—places competitors on unequal footings. Significance Statement Until recently, weather forecasts have relied on physics-based numerical weather prediction (NWP) models. Since late 2022, the advent of artificial-intelligence-based weather prediction (AIWP) models that learn exclusively from data, without the laws of physics being invoked, challenges extant practice. While AIWP models might eventually supersede NWP models, meaningful comparisons of forecast quality remain difficult due to differences in training schemes, initial conditions, and postprocessing. Our paper introduces a new measure, called potential continuous ranked probability score (PCRPS), that accounts for these differences and, therefore, affords fair comparisons. Using benchmark data, we confirm the superiority of AIWP models for gridpoint-based global forecasts of surface weather variables at lead times up to 10 days.
Language: English
Open Access status: green
OpenAlex ID: W7117464362
FWCI: 2.146
Cites: 38
Cited by: 7
Related to: 10