Work

Conformal Expert Pools: Training-Free Probabilistic Time-Series Forecasting Beyond Seasonality

Year: 2026
Type: preprint
Abstract: Conformal Seasonal Pools (CSP) showed that a forecaster with no training at all, built only from past values at the same point in the season and from past forecast errors, significantly outperformed DeepNPTS, Amazon's learned non-parametric forecaster. CSP was the more accurate of the two on 66% of 232 datasets, its 95% prediction intervals contained the outcome 90.1% of the time against 57.8% for DeepNPTS, and DeepNPTS was 819 times slower. CSP was always meant to be the simplest possible probabilistic forecasting baseline, a few lines of numpy code, and was never designed as a full-fledged forecasting model. It forecasts the level of a series in one way only, with the seasonal-naive forecast, and so leaves room for improvement on trending series, short histories, changing volatility and series with many zeros. We propose Conformal Expert Pools (CEP), a training-free successor. CEP replaces the single seasonal-naive forecast by a pool of simple forecasters, builds its predictive distribution from their past forecast errors, and adapts both the pool and the errors to the series being forecast. No model is trained and no parameter is learned across series. We evaluate CEP on the CSP benchmark of 232 datasets and 24,502 forecast windows, using 95% prediction intervals. We report two main probabilistic scores, for both of which lower is better: CRPS, which measures the accuracy of the whole predictive distribution, and the scaled interval score (MSIS), which penalises intervals that are wide or that miss the outcome. Compared with CSP, CEP is better on 78% of datasets by CRPS, on 84% by MSIS and on 83% by the unscaled Winkler score, and on 78% by the mean quantile loss used in the GluonTS and DeepNPTS evaluations. On a typical dataset its CRPS is 14% lower and its MSIS 23% lower (both p < 1e-22, paired Wilcoxon test), and interval coverage rises from 90.1% to 92.2%. CEP also has a lower CRPS than DeepNPTS on 88% of datasets, than NPTS on 89% and than SeasonalNPTS on 83%. The improvement over CSP persists on the datasets from families outside the development set (better on 76% by CRPS, 83% by MSIS and 81% by Winkler score) and on 36 additional files from fev-bench outside the benchmark (78%, 81% and 81%). Point forecasts improve as well: median MASE falls from 1.10 to 1.01 and median RMSSE from 0.94 to 0.87. Finally, we compare CEP with Laplace, an online ensemble forecaster that borrowed from CSP: by its author's account, after losing to CSP on hourly data it adopted a "seasonal anchor", which blends a recency-weighted average of past values at the same point in the season, the idea behind CSP's seasonal pool, with the seasonal-naive value on which CSP centres its residuals. Its author states that "Laplace beats (almost) everything" and reports that it "beats" CSP on most series in three of four test collections. Those claims rest on a narrow test: one-step-ahead forecasts only, on a limited and unrepresentative set of series chosen by Laplace's author (economic series from one source and one hourly collection), with no interval widths or interval scores reported. We tested them on the CSP benchmark, on 2,534 multi-step windows from 228 datasets with the same history given to every method, and they do not hold on large-scale studies such as this one. The first claim is false: plain CSP outperforms Laplace on 60% of datasets by CRPS and on 64% by interval score, and its point forecasts are more accurate on 56% by MASE. Scored alongside all the methods of the benchmark on these datasets, Laplace ranks sixth of seven on both CRPS and interval score: below SeasonalNPTS on both, below DeepNPTS on CRPS, and with by far the widest intervals of all seven (69% wider than CEP's). Laplace's author did not report interval widths, although it is generally accepted that a probabilistic forecast is judged on sharpness (interval width) together with calibration (coverage); Laplace's intervals are 44% wider than CSP's. Its weakness is at multi-step horizons: from 13 to 96 steps CSP has the lower CRPS on 65% of datasets. Its point forecast is biased low (bias -0.24 in scaled units, against -0.02 for CEP and CSP) and on 54% of datasets is less accurate than the seasonal-naive forecast. Absence of bias is a key requirement of any forecaster: a forecast that is systematically too low means stock-outs in demand planning and under-procurement for an energy supplier bidding into the grid. Laplace fails this requirement. It also produces pathological forecasts: on electricity load it abandons the daily cycle and forecasts negative demand within a day, in 7.6% of forecasts of non-negative series its median goes below zero, and on some series its forecast distribution becomes numerically unstable, with intervals fifteen orders of magnitude wider than the data and CRPS more than ten times CSP's on 9 datasets. From a cold start Laplace takes about 7,700 times longer than CSP, from 2,500 times at horizons up to 12 to 24,000 times beyond 96, where a single forecast takes 107 seconds, and the longest horizons cannot be run in practice. CEP in turn is better than Laplace on 82% of datasets by CRPS, on 80% by MSIS and on 82% by Winkler score, its intervals are about 40% narrower, and it runs 271 times faster.
Language: English
Open Access status: green
OpenAlex ID: W7228826033
Cites:
Cited by:
Related to: 10