How Far Does Zero-Shot Go? A Leakage-Aware Evaluation of TimesFM 3 on Post-Release, Multivariate and Controlled Data
Year: 2026
Type: preprint
Abstract: Preprint, not peer reviewed. TimesFM 3, released by Google in August 2026, is reported to lead the public forecasting benchmarks, whose data are old and public. We evaluate it zero-shot under a protocol frozen before the post-release test data were downloaded: on real data published after its release (electricity load and prices, weather, river flow, exchange and interest rates), on a Wikipedia probe that compares the same pages inside and outside the declared period of its training corpus, on two prospective issues sealed by hash before their outcomes existed, and on 7,600 controlled synthetic tasks; we also reproduce Google's fev-bench results and read the inference code. On a 30-task sample the published results largely reproduce. On the four structured post-release domains TimesFM 3 has a 16% lower MASE and a 29% lower scaled quantile loss than a pre-specified classical ensemble; the size depends on the baseline and the weighting. Its lead over Chronos-2, about 5%, depends on context. In two overlapping prospective weeks the quantile-loss advantage over the ensemble is confirmed, but on load, prices and river flow the point-forecast advantage is not, and the nominal 80% intervals cover about 70% of the outcomes. On exchange rates, interest rates and oil prices TimesFM 3 is worse than the random walk; in synthetic data it fails on a saturating process and after recent upward level shifts. The Wikipedia probe does not identify an effect of memorisation. The zero-shot advantage depends on the structure of the series, the information given and the loss, and the reliability of the quantiles needs checking separately from their average accuracy. Code, frozen protocol, metrics, forecasts, data and the public timestamps of the prospective seals are released (see related works). TimesFM 3 weights are not redistributed.
Author: Pomaro, Alessio
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225989732
