Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Year: 2025
Type: article
Study design: Observational Study
Abstract: The growing popularity of generative AI tools presents new challenges for data quality in online surveys and experiments. This study examines participants’ use of large language models to answer open-ended survey questions and describes empirical tendencies in human vs LLM-generated text responses. In an original survey of participants recruited from a popular online platform for sourcing social science research subjects, 34% reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants’ LLM use in online research.
Source: Sociological Methods & Research
Authors: Simone Zhang, Janet Xu, AJ Alvero
Institutions: New York University, Stanford University, Cornell University
Language: English
Subfield: Sociology and Political Science
Field: Social Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $3,900
OpenAlex ID: W4410176054