Experiment-based calibration in psychology: Foundational and data-generating model
Year: 2025
Type: article
Abstract: Experiment-based calibration is a novel method for measurement validation, which unlike classical validity metrics does not require stable between-person variance. In this approach, the latent variable to be measured is manipulated by an experiment, and its predicted scores - termed standard scores - are compared against the measured scores. Previous work has shown that under plausible boundary conditions, the correlation between standard and measured scores - termed retrodictive validity - is informative about measurement accuracy, i.e. combined trueness and precision. Here, I expand these findings in several directions. First, I formalise the approach in a probability-theoretic framework with the concept of a standardised calibration space. I show that previously derived boundary conditions hold for arbitrary distributions of standard, true, and measured scores. Second, I relate this framework to classical validity theory and show that the boundary conditions in fact apply to any form of criterion validity, including classical convergent validity. Thus, I state precise and empirically quantifiable boundary conditions under which criterion validity metrics are informative on validity. Third, I relate these boundary conditions to confounding variables, i.e. correlated latent variables. I show that in the limit, calibration will converge on the latent variable that is most closely related to the standard. Finally, I provide a framework for modelling the data-generating process with Markov kernels, and identify sufficient conditions under which the data generation model results in a calibration space. In sum, this article provides a formal probability-theoretic framework for experiment-based calibration and facilitates modelling and empirical assessment of the data generating processes.
Author: Dominik R. Bach
Language: English
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,440
OpenAlex ID: W4414246044