Effect estimates using publicly available school-level data in a cluster-randomized educational experiment
Year: 2025
Type: conference-paper
Abstract: Effect estimates from randomized trials, though famously free of confounding bias, may nevertheless be too noisy to be of much use. Recent work has shown that supplementing experimental analyses with observational data can improve statistical precision without contributing confounding bias. The idea is, basically, to use observational data to train an algorithm to use baseline covariates to predict experimental outcomes, and then to use predicted outcomes as an additional covariate in effect estimation. However, due to student privacy regulations, observational data is often unavailable. Moreover, outcomes for many educational field trials use bespoke learning measures that are not available in analogous observational datasets. This paper illustrates a way to circumvent these issues in the context of the Texas subset of the Cognitive Tutor Algebra 1 effectiveness trial, which features school-level randomization and a specialized posttest. We train a model using publicly available school-level covariates to predict state-test school passing rates, and use the resulting school-level predictions to reduce the standard errors in a student-level hierarchical effect model.
Authors: Adam Sales, Charlotte Mann, Johann Gagnon-Bartsch, Neil Heffernan
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W6949817834
FWCI:
Cites:
Cited by:
Related to: