Evaluating intermittent demand forecasts: a preregistered study on four public datasets
Year: 2026
Type: preprint
Abstract: A first note on one dataset, and its review, suggested four things about intermittent demand forecasts: that per-period error measures reward a forecast of zero, that the point forecast under a quantile matters more than the distribution placed around it, that a model trained across all series is less biased than per-series smoothing without being more accurate, and that per-period accuracy does not rank methods for stock. We turned them into hypotheses with decision rules, chose every setting on development data, and deposited the protocol and the hashes of the code publicly before reading the test windows of four public datasets (Auto, RAF, M5, UCI Online Retail) once. A forecast of zero has the lowest mean MASE of all methods on the three datasets where most periods are zero. The pooled gradient-boosting model is not more accurate than the best smoothing method by more than 1 % on any of the four datasets. We find no evidence that it is less biased: on M5 it is more biased than the method it was compared with, and on the three other datasets the test cannot detect a gap of 5 %, so there is no conclusion. Whether the base matters more than the distribution depends on the dataset and on the level: confirmed on two datasets at level 0.90, refuted on one at 0.90 and on three at 0.95. Accuracy and stock rank the methods differently on RAF and largely agree on M5. One statement holds on every dataset (the pooled model is not more accurate), one holds for MASE but not for RMSSE (zero is rewarded), and three do not hold as general statements (base against distribution, lower bias of the pooled model, accuracy against stock). Protocol, code, settings and results are public.
Author: Chadeffaud, Mahé
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7233566946
Cites:
Cited by:
Related to:
