Examining large language models for safety and robustness through the lens of social science
Year: 2025
Type: article
Abstract: Large language models have demonstrated remarkable capabilities, often achieving human-like performance levels and significantly impacting our daily lives.However, these models can perpetuate and amplify harmful stereotypes and biases associated with socio-demographic representations, potentially generating discriminatory content that adversely affects individuals and communities.Given these risks and their broader societal implications, ensuring the safety and robustness of these models through the identification and mitigation of harmful stereotypes has become imperative.This dissertation presents comprehensive methodologies to address these challenges by integrating insights from social science, psychology, and cognitive studies with methods from natural language processing.First, we present a framework to assess human-like stereotypical patterns in large language models (LLMs), drawing upon established psychological theories of how individuals develop stereotypes toward various social groups.This theoretically-grounded approach provides construct validity in defining and measuring stereotypes.The framework incorporates three key dimensions: warmth-competence analysis, keywordreasoning patterns, and emotional-behavioral responses.Through clustering analysis, keyword extraction, and reasoning pattern evaluation of LLM responses, we examine how these models align with or deviate from documented human behavioral patterns.Our findings reveal that LLMs demonstrate nuanced perceptions of social groups, consistent with psychological research highlighting the multifaceted nature of stereotypes.Notably, the models' reasoning patterns, particularly regarding groups' economic status, demonstrate a nuanced awareness of societal disparities.Second, we propose methods to examine causal sensitivity of language models on socio-demographic attributes.This is based on a controlled experimental framework that uses name frequency analysis from U.S. Census data and systematic evaluation of model predictions through causal graphs.Our findings show that less frequent first names lead to divergent model predictions, highlighting the need for careful demographic consideration in dataset design to ensure fair and consistent model performance across different name representations.Third, we investigate how LLMs exhibit and inflate political stereotypes through the lens of cognitive biases and representative heuristics.We analyze LLMs' responses using two key theoretical frameworks: 'kernel of truth' (whether stereotypes reflect empirical realities) and 'representative heuristics' (whether models overemphasize representative attributes of target groups), comparing model outputs with actual human responses across various political topics.Our findings show that while LLMs can accurately mimic certain political positions, they tend to exaggerate these positions compared to empirical human responses, suggesting a vulnerability to stereotypical thinking similar to human cognitive biases.This implies the need for careful consideration of cognitive bias frameworks in developing and deploying language models, 4.3 The disaggregatd version of Anes responses with averaged scales. . . . . . . . . . . . . . . . . .51 4.4 The difference between the MFQ average responses and the Empirical Mean is shown.For most models, the difference is positive for Republicans, indicating an inflating tendency, with exceptions like Llama3-8b, which shows negative differences on several traits.For Democrats, the difference is negative, suggesting a deflating tendency, although some exceptions are present. . . . . . . . . . . .52 4.5 The x-axis corresponds to the Empirical Mean Difference (E(a|X + ) -E(a|X -)), y-axis corresponds to the Believed Mean Difference (E B (a|X + ) -E B (a|X -)) of each question.The black line indicates y = x.53 x Chapter 1
Author: Jeoung, Sullam
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: green
OpenAlex ID: W7220826462
