Work

Understanding Social Biases in Large Language Models

Year: 2025

Type: article

Abstract: Background/Objectives: Large Language Models (LLMs) like ChatGPT, LLAMA, and Mistral are widely used for automating tasks such as content creation and data analysis. However, due to their training on publicly available internet data, they may inherit social biases. We aimed to investigate the social biases (i.e., ethnic, gender, and disability biases) in these models and evaluate how different model versions handle them. Methods: We instruction-tuned popular models (like Mistral, LLAMA, and Gemma), and for this we curated a dataset constructed by collecting and modifying diverse data from various public datasets. Prompts were run through a controlled pipeline, and responses were categorized (e.g., biased, confused, repeated, or accurate) and analyzed. Results: We found that models responded differently to bias prompts depending on their version. Fine-tuned models showed fewer overt biases but more confusion or censorship. Disability-related prompts triggered the most consistent biases across models. Conclusions: Bias persists in LLMs despite instruction tuning. Differences between model versions may lead to inconsistent user experiences and hidden harms in downstream applications. Greater transparency and robust fairness testing are essential.

Source: AI

DOI: 10.3390/ai6050106

Authors: Ojasvi Gupta, Stefano Marrone, Francesco Gargiulo, Rajesh Jaiswal, Lidia Marassi

Institutions: Technological University Dublin, University of Naples Federico II, National Research Council

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDG: Reduced inequalities

Open Access status: gold

Estimated APC paid: $1,930