Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing
Year: 2023
Type: conference-paper
Abstract: Brand Lift test is a widely deployed statistical tool for measuring the effectiveness of online advertisements on brand perception such as ad recall, brand familiarity and favorability. By formulating the problem of interest into a two-sample test on the binomial proportions from the control group (p0) and the treatment group (p1), Brand Lift test evaluates ads impact based on the statistical significance of test results. Traditional approaches construct the test statistics based on the absolute difference between the two observed proportions, a.k.a, absolute lift. In this work, we propose a new bootstrap test based on the percentage difference between the two observed proportions, i.e., relative lift. We provide rigorous theoretical guarantees on the asymptotic validity of the proposed relative-lift-based test. Our numerical studies suggest that the relative-lift-based test requires less stringent conditions than the absolute-lift-based test for controlling the type-I error rate. Interestingly, we also prove that the relative-lift-based test is more powerful than the absolute-lift-based test when the alternative is positive (i.e., p1-p0>0), but less powerful when the alternative is negative (i.e., p1-p0<0). The empirical performance of the proposed test is demonstrated by extensive simulation studies, an application to a publicly available A/B testing dataset from advertising, and real datasets collected from the Brand Lift Testing platform at LinkedIn.
Institutions: LinkedIn (United States), University of Notre Dame
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4387846339