Work

Transcribing and Coding Voice Answers Obtained in Web Surveys: Comparing Three Leading Automatic Speech Recognition Tools

Year: 2025

Type: article

Abstract: With the rise of smartphone use in web surveys, voice or oral answers have become a promising methodology for collecting rich data. Voice answers present both opportunities and challenges. This study addresses two of these challenges—labor-intensive manual transcription and coding of responses. We compare the transcription performance of three leading Automatic Speech Recognition (ASR) tools—Google Cloud Speech-to-Text API, OpenAI Whisper, and Vosk—using voice answers collected from an open-ended question on nursing home transparency that was administered in an opt-in online panel in Spain. Additionally, we evaluate the efficiency and quality of coding these transcriptions using human coders and GPT-4o, a Large Language Model (LLM) developed by OpenAI. We found that each of the ASR tools has distinct merits and limits. Google sometimes fails to provide transcriptions, Whisper produces hallucinations (false transcriptions), and Vosk has clarity issues and high rates of incorrect words. Human and LLM-based coding also differ significantly. Thus, we recommend using several ASR tools for voice answer transcription and implementing human as well as LLM-based coding, as the latter offers additional information at minimal added cost.

Source: Journal of Survey Statistics and Methodology

DOI: 10.1093/jssam/smaf028

Authors: Melanie Revilla, Carlos Ochoa, Jan Karem Höhne, Mick P Couper

Institutions: Universitat Pompeu Fabra, Leibniz University Hannover, German Centre for Higher Education Research and Science Studies, University of Michigan, Institute for Social Research

Language: English

Topic: Survey Methodology and Nonresponse

Subfield: Sociology and Political Science

Field: Social Sciences

Domain: Social Sciences

Open Access status: hybrid

Estimated APC paid: $3,969

Funder: H2020 European Research Council