Work

When do e-commerce prediction models fail over time? cross-task evidence on temporal reliability and model maintenance

Year: 2026
Type: article
Abstract: Machine-learning models in e-commerce are often evaluated on randomly partitioned historical data, whereas deployment requires prediction on future observations under changing populations and information availability. We examine temporal reliability and model maintenance across four public dataset–task settings: late-delivery prediction among eventually delivered Olist orders, matched 24-h cart-to-purchase prediction in Retailrocket and Synerise RecSys 2025, and 30-day repeat-purchase prediction in Online Retail II. Sequential analyses restrict historical features, model updating, and recalibration to information and labels available at each prediction or deployment time. Random-to-future differences varied substantially: Olist showed pronounced deterioration, whereas the other settings showed smaller or qualitatively different changes. In the primary Olist XGBoost analysis, cumulative retraining improved ranking and modestly improved Brier accuracy, while sliding-window retraining improved ranking but worsened probability reliability. Cross-model, matched-exposure, and sensitivity analyses showed that maintenance effects depended on setting, period, model family, and design. Post-hoc recalibration did not provide a consistently reliable repair, and same-base calibration effects differed from net maintenance-policy effects. The matched comparisons do not identify causal task or duration effects. These findings support forward-in-time evaluation that assesses ranking, probability quality, label availability, and maintenance effects separately rather than inferring future reliability from offline performance or task identity.
Language: English
Subfield: Marketing
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7232010490
FWCI:
Cites: 22
Cited by:
Related to: 10