Work

Assessing self-selection biases in online surveys: evidence from the COVID-19 Health Behavior Survey

Year: 2023

Type: report

Abstract: During the COVID-19 pandemic, many primary data collection efforts relied on online surveys via social media recruitment.According to the leveragesalience theory, respondents' differential levels of interest in the survey topic can lead to differential survey responses, potentially introducing biases.In this study, we investigate the potential impact of survey recruitment materials on survey responses.We use data from the "COVID-19 Health Behavior Survey", a cross-national online survey that we ran between March and August 2020 in eight countries in Europe and North America (N=120,184).Respondents were recruited via targeted Facebook advertisements with varying degrees of reference to the survey topic of COVID-19.The aim of our study is to understand whether survey responses from images more strongly (or weakly) related to COVID-19 are associated with higher (or lower) threat perceptions of COVID-19 and the adoption of preventive behaviors, including face mask usage and increased hand-washing.Descriptive results indicate variations in our sample composition by ad image in terms of sex, age, and educational status of respondents.Our findings from the regression analyses show that, in 18 of the 32 models, ad images had no significant effect on the survey outcomes.Factors like the month of survey participation or respondents' age were more influential.In the remaining models where unexplained image effects persisted, the impact was minimal.The mask-wearing images were generally associated with lower threat perceptions of COVID-19 to oneself and the family.On the contrary, we found no consistent association between higher topic salience and the adoption of protective behaviors.Our findings provide empirical evidence of minimal self-selection biases based on recruitment images used early on during the COVID-19 pandemic, thus alleviating concerns regarding potential sources of bias in unobservable characteristics.

DOI: 10.4054/mpidr-wp-2023-047

Authors: Jessica Donzowa, Daniela Perrotta, Emilio Zagheni

Institutions: Bielefeld University, Max Planck Institute for Demographic Research

Language: English

Topic: Survey Methodology and Nonresponse

Subfield: Sociology and Political Science

Field: Social Sciences

Domain: Social Sciences

SDG: Good health and well-being

Open Access status: gold

Funders: International Max Planck Research School for Environmental, Cellular and Molecular Microbiology, Max-Planck-Gesellschaft, International Max Planck Research School for Advanced Methods in Process and Systems Engineering, Max-Planck-Institut für demografische Forschung

OpenAlex ID: W4388682604

Cites: 36

Cited by: 6

Related to: 10