When Does Uplift Modelling Pay? A Correlation Criterion and a Measurement Floor for Small-Scale Subscription Retention
Year: 2026
Type: article
Abstract: Uplift modelling — targeting customers by estimated treatment effect rather than byoutcome propensity — is widely recommended and inconsistently useful. We show thatwhether it pays is governed by a single measurable quantity: corr(τ̂, π̂), thecorrelation across customers between the estimated treatment effect and the estimatedoutcome propensity. Across four real randomised controlled trials (Hillstrom, split byarm; Criteo-UPLIFT v2.1; Lenta) and one calibrated simulator, the advantage ofeffect-based targeting rises monotonically as this correlation falls: it is -5.6% atcorr = +0.69, +0.6% at +0.58, +12.7% at +0.19, +20.3% at +0.17, and +106.9% at -0.19.Subscription retention is the adversarial case, and the mechanism is absent fromadvertising: the customers a churn model ranks highest include dormant payers for whombeing contacted is itself the reminder to cancel. We record a prediction made before theLenta data was obtained — that retail promotion would fall between advertising andretention — which landed as predicted. We then show this diagnosis is of limited use to the businesses it most concerns,because at their scale neither method is reliable. Holding the evaluation set fixed andshrinking only the training set on a real 64,000-customer experiment, the besteffect-based method beats random targeting on only 75% of draws at n = 500; one standardestimator manages 55%. Mean performance at that size looks respectable; a business getsone draw. We propose a money-denominated decision layer that can abstain, and report its partialfailure honestly: it beats a ranking policy on 93% of draws but does not beat doingnothing, and a per-customer offer optimiser beats a well-chosen single offer on 58% ofdraws, CI [0.42, 0.72] — indistinguishable from chance. Two units errors found in our owncode are reported as findings, one of which every one of 337 then-passing tests failed todetect because a units error is self-consistent. Finally, we build and validate the measurement layer. A holdout-based incrementalityestimator recovers the simulator's known effect with bias centred on zero and 88–98%interval coverage — and its minimum detectable effect exceeds the delivered effect atevery business size tested, including 10,000 customers. Detecting an effect of 0.0108 at80% power with a 10% holdout requires approximately 119,500 customers. A smallsubscription business therefore cannot measure its own retention campaigns, which weargue is a mechanism for the industry's reliance on uninterpretable save-rate reporting.
Authors: GUPTA, RISHI, Jain, Prasham
Institution: Manipal University Jaipur
Language: English
Subfield: Marketing
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7203771157
FWCI:
Cites:
Cited by:
Related to: