Enhancing Thematic Analysis with Large Language Models: A Comparative Study of Structured Prompting Techniques
Year: 2024
Type: dissertation
Abstract: Thematic analysis, while crucial for qualitative research, becomes challenging with large datasets due to the intensive manual effort involved. This thesis explores the capacity of Large Language Mod- els (LLMs) to automate and enhance this process through different prompting techniques, benchmarking their performance against human-generated analyses. In this thesis we compare three distinct approaches: (1) Zero-Shot prompting, a single-prompt solution relying solely on pre-trained knowledge; (2) Top- Down Structured Prompting (TDSP), leveraging prompt engineering principles by starting with broad themes and progressively refining analysis; and (3) Bottom-Up Structured Prompting (BUSP), mirroring the established six steps of human thematic analysis designed for efficient extraction of nuanced insights. Applying these techniques to three diverse qualitative datasets, we used a rubric-based scoring system to evaluate the outputs against findings from the original research. Our results demonstrate that structured prompting techniques, particularly TDSP, significantly outperform the Zero-Shot approach in accurately capturing key findings. On average, 86% of findings identified by human researchers were at least par- tially identified in the outputs generated using structured prompting techniques. We also explored using an AI-based evaluator for rubric scoring but found the results unreliable, indicating that human evalua- tion remains crucial. This study makes a compelling argument that AI can efficiently perform thematic analysis and, especially in the future, provide substantial support to qualitative researchers in their work.
Author: Wittmann, Ferdinand
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7037593071