In AI Do We Trust? Comparing a Transformer-Based Natural Language Processing Model Against a Traditional Content Analysis Approach to Qualitative Research
Year: 2026
Type: preprint
Abstract: Central to debates about the use of artificial intelligence (AI) models for qualitativeresearch are concerns about comparability and trust in AI generated findings. The presentmethodological study compared findings from a transformer-based Natural Language Processing(NLP) model, BERT, against a traditional content analysis approach to qualitative research usingthe same text data. We examined claims that AI models generate findings that are identical orsimilar to traditional qualitative approaches. The study assessed three aspects of comparabilitybetween the BERT model solution and human-coded findings that included topic labeling,classification overlap, and AI replicability. Results indicated broad similarity in the semanticmeaning of topic categories generated by the BERT model and themes from content analysis.There was low overlap in the classification of individual participant responses between BERTand content analysis categories, even for categories similar in semantic meaning. Running theBERT model a second time on the same input data resulted in differences in topic categories anddocument classification between the two approaches. Study implications and researchrecommendations for the use of NLP models are discussed.
Source: PsyArXiv (OSF Preprints)
Authors: Matthew Baron, Catherine Stein
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7221164480