Wellbeing web survey
Year: 2022
Type: study-registration
Abstract: As in previous Registrations, the foremost intended output from the Baseline survey (and related surveys) is a paper which is tentatively titled, “Adjusting for Scale-Use Heterogeneity in Self-Reported Well-Being.” The Baseline survey consists of four types of questions: aspect ratings, aspect tradeoffs, calibration questions, and “other” questions. The “other” questions fall in five general categories: demographic questions, questions to learn about respondents’ scale-use tendencies, questions to explore respondents’ interpretation of specific aspects, questions to assess respondent effort and experience, and exploratory questions. FIELDING HISTORY We fielded an exploratory “N=500” sample of the Baseline survey on Amazon MTurk in June-August 2020. We have also fielded follow-up surveys (Baseline2 and BottomlessA) on that sample. We have used these data to further refine our methodology for using calibration questions to correct self-reported well-being ratings for interpersonal differences in scale use. We made minor edits to the Baseline survey and fielded a larger “N=4000” sample on MTurk in June-July 2022. We implemented our planned quality-control procedure (using open-ended text box questions) and found, unfortunately, that the survey responses were of lower quality than in 2020. Of 3,309 respondents who completed the survey, only 976 (29.5%) passed quality control and qualified for the primary analysis sample and follow-up survey invitations. In the “N=500” sample from 2020, the counterpart figures had been 292 and 497, respectively, for a qualification rate of 58.8%. In light of the lower response quality, we made two changes to the survey design. We added a brief 20-question “Prescreening” survey before the Baseline, which includes the text box questions, so that we pay less compensation to respondents who don’t pass quality control. We planned to begin recruiting survey participants on Prolific (a different crowdsourcing labor platform) in addition to Amazon's Mechanical Turk (MTurk). We hoped that Prolific respondents pass quality control at a higher rate (based on other researchers’ anecdotal experiences and some Prolific features—they require a photo ID for registration and restrict IP addresses from VPNs). Apart from quality issues, Prolific also should help us get to our target sample size, which is N=4000 after quality control. In July 2022, we fielded a sample of N=100 respondents on MTurk using a first draft of the Prescreening survey which was ultimately unsuccessful. We wanted to avoid duplicating respondents across survey platforms, so we asked respondents if they had completed the Baseline survey before (as described by the MTurk HIT title and a screenshot). 64% of respondents indicated that they had taken it, even though zero of their MTurk worker ID’s appeared in our previous data. Given this very low response quality, we will not use this question again going forward. THIS REGISTRATION We are updating our quality control strategy, as described in the narrative summary and documents. We will consider the first batch of respondents who completed the Prescreening survey as a test batch only, and not invite them to the Baseline or other surveys. Our next step is to field an updated Prescreening survey on MTurk with N=100 respondents, followed by the Baseline and other surveys according to our quality control procedure.
Source: Open Science Framework
Language: English
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7171319654