Work

Weighting the weeks that matter: entropic tilting for epidemic forecast evaluation

Year: 2026
Type: preprint
Abstract: The performance of infectious disease forecasting models is commonly calculated under the implicit assumption that forecast accuracy is equally important across time. This approach may obscure or under-weight differences in model performance during the epidemic periods that matter most for public-health decisions. Here, we introduce the weighted prequential score (WPS), a forecast evaluation framework that can emphasize epidemic periods of interest. The WPS is a weighted average of scores over time, where weights are constructed through entropic tilting which up-weights priority periods while keeping the WPS as close as possible to the arithmetic average. We demonstrate the WPS on forecasts of incident hospitalizations submitted to the CDC FluSight challenge, focusing on two priority periods: weeks of rapid growth in incidence and weeks near the seasonal peak. Across season-location-horizon pairs, doubling the weight assigned to priority periods changes the top-ranked model in 10%-26% of eligible settings, while the top-ranked model remains unchanged regardless of the amount of weight assigned in 37%-68% of settings. We demonstrate that for suitable weights, the models favored by the WPS yield an average gain in accuracy during priority periods that is at least as large as the average accuracy lost during the remaining weeks. Because weights can be defined using observed outcomes, the WPS is not a proper scoring rule. Hence, we recommend using the WPS as a retrospective evaluation diagnostic. The WPS can guide public-health practitioners in selecting models that perform well during periods most relevant to their decisions, while encouraging forecasting teams to improve accuracy during these critical weeks.
Source: medRxiv
Language: English
Subfield: Epidemiology
Field: Medicine
Open Access status: green
OpenAlex ID: W7233548555
Cites:
Cited by:
Related to: