Year: 2015
Type: dissertation
Study design: Observational Study
Abstract: The applied nature of higher education assessment does not lend itself to rigorous experimental research designs. However, assessment practitioners would like to make claims about the influence of educational programs on student learning outcomes. Propensity score matching (PSM) methods are quasi-experimental techniques that allow researchers to control for known confounding variables. In the context of higher education, PSM techniques allow assessment practitioners to control for confounding variables related to students’ self-selected participation in university programs. Research and recommendations on how to apply PSM techniques are scattered throughout several disciplines. However, additional research is needed to evaluate how well PSM techniques control self-selection bias in the context of educational assessment. To couch PSM techniques within the framework of higher education assessment, the current study first summarized common practices and recommendations from literature across several disciplines, then evaluated the application of common PSM techniques via an applied example of honors program assessment. Specifically, the study applied common PSM techniques to compare students in the honors program with students either not invited into the program or students who decided not to participate in the program. Data analyses four research questions: 1) Do honors students differ from students not in the honors program on motivation variables?; 2) How well do different common PSM techniques create quality comparison groups of students?; 3) How well do different common PSM techniques retain honors students in the comparison of program outcomes?; and 4) Do honors students differ from students not in the honors program on outcomes after PSM techniques are applied? Honors students did not differ from students not in the honors program on motivation variables thought to be related to self-selection into the program. Across matching methods applied in the current study, the quality of propensity score matches was near optimal. Average scores on the outcomes of interest to the honors program did not significantly differ by group. However, decreased sample sizes resulted in a loss of minority student representation in the honors sample and different practically significant results on program outcomes. Recommendations and implications for applied assessment practitioners are offered.
Author: Heather D. Harris
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W777002935