Understanding Predictive Models of Student Success with a Multiverse Analysis
Year: 2025
Type: conference-paper
Abstract: Predictive models of student success can provide timely information to inform interventions in K-12 and higher education. However, the design and implementation of these predictive models require various stakeholders to make decisions about the prediction target, data sources, processing, training, models, and deployment strategies. These choices are often poorly documented in the scholarly literature, even when code is openly available, limiting our ability to generalize and translate research findings to other institutions or contexts. More importantly, it obfuscates the potential trade-offs of decisions that are made with respect to prediction performance and other objectives, such as group fairness criteria. To address these challenges, we advocate for a multiverse approach in student success modeling and demonstrate the approach using a case study. In the multiverse framework, each plausible choice made to refine the problem space results in separate analyses being completed (each being referred to as a "universe"), with the final result being the collection of all universes explored. We demonstrate the mechanics and merits of this approach by building a first-year retention model for higher education. We interpret the findings of this analysis, specifically considering both model goodness-of-fit and fairness by group, demonstrating the value of the multiverse technique in engaging education-specific stakeholders—from administrative supervisors to model developers—in making predictive models that are robust, reproducible, and equitable.
Authors: Yunxuan Tang, Emma Harvey, Chengyuan Yao, Renzhe Yu, Rene Kizilcec, Christopher Brooks
Language: English
Subfield: Computer Science Applications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6893329276
FWCI:
Cites:
Cited by:
Related to: