Work

Retrospective pre/posttest design and response-shift bias in an urban after-school program for teens: A mixed methods study

Year: 2016
Type: article
Abstract: Evaluators need more design options to meet the challenges they face in detecting change or growth. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects, but few studies have used this approach with youth. After School Matters, a Chicago nonprofit that provides after-school programs to teens tested the retrospective pretest/posttest design using a mixed methods design to determine whether response-shift bias exists. My study provided several findings. First, though my quantitative analysis did not indicate response-shift bias was as prevalent as literature would indicate, my qualitative findings indicated that response-shift bias was in fact an issue. Second, I found a relationship between teens’ self-reported interpersonal skills and response-shift bias. Teens who reported positive interactions with their peers and instructors tended to display large shifts in their responses from traditional pretest to retrospective pretest. Third, teens preferred to see the posttest and retrospective questions in chronological order, which is contrary to the literature. Fourth, I found acquiescence to be the biggest potential bias when using the design with teens. Overall, the retrospective pretest/posttest design is a practical and useful design to evaluate youth self-reported change. The mixed methods design led to dissonance, iterative data analysis, and some inconclusive findings, but also a much deeper understanding of response-shift bias.
Author: Jill Young
Language: English
Open Access status: green
OpenAlex ID: W2582395487
FWCI:
Cites: 22
Cited by: 3
Related to: 10