Incremental validity is a statistically problematic concept
Year: 2015
Type: article
Abstract: Psychologists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte Carlo simulations, and a novel analytical framework to demonstrate that two widespread strategies for establishing incremental construct validity using multiple regression analysis exhibit extremely high Type I error rates under parameter regimes common in many psychological domains. Counterintuitively, we find that error rates are highest—in some cases approaching 100%—when sample sizes are large and reliability is moderate. Our findings suggest that a potentially large proportion of incremental validity claims made in the literature are spurious. We present a web application (http://jakewestfall.org/ivy/) that readers can use to explore the statistical properties of these and other incremental validity arguments. We conclude by discussing appropriate statistical methods for establishing incremental validity.
Source: Figshare
Authors: Yarkoni, Tal, Westfall, Jacob
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6901962969