Year: 2024
Type: conference-paper
Abstract: Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses.However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question.This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis.Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics.However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis.1 * Equal contribution.
Language: English
Subfield: General Social Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
Awards: THE CULTURAL, ECONOMIC, AND INSTITUTIONAL DETERMINANTS OF AI INFRASTRUCTURES AND THEIR CONSEQUENCES IN GLOBAL CONTEXTS, MRI: Acquisition of an Adaptive Computing Infrastructure to Support Compute- and Data-Intensive Multidisciplinary Research, EAGER: Building Language Technologies by Machine Reading Grammars
OpenAlex ID: W4404782249