The Elusiveness of Detecting Political Bias in Language Models
Year: 2024
Type: conference-paper
Abstract: This study challenges the prevailing approach of measuring political leanings in Large Language Models (LLMs) through direct questioning. By extensively testing LLMs with original, positively and negatively paraphrased Political Compass questions we demonstrate that LLMs do not consistently reveal their political biases in response to standard questions. Our findings indicate that LLMs' political orientations are elusive, easily influenced by subtle changes in phrasing and context. This study underscores the limitations of direct questioning in accurately measuring the political biases of LLMs and emphasizes the necessity for more refined and effective approaches to understand their true political stances.
Institution: University of Udine
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4403577462