Evaluating LLMs for Detecting Demographic-Targeted Social Bias: A Comprehensive Benchmark Study
Year: 2026
Type: conference-paper
Abstract: Large-scale web-scraped text corpora used to train general-purpose AI models often contain harmful demographictargeted social biases, creating a regulatory need for data auditing and developing scalable bias-detection methods.Although prior work has investigated biases in text datasets and related detection methods, these studies remain narrow in scope.They typically focus on a single content type (e.g., hate speech), cover limited demographic axes, overlook biases a!ecting multiple demographics simultaneously, and analyze limited techniques.Consequently, practitioners lack a holistic understanding of the strengths and limitations of recent large language models (LLMs) for automated bias detection.In this study, we conduct a comprehensive benchmark study on English texts to assess the ability of LLMs in detecting demographic-targeted social biases.To align with regulatory requirements, we frame bias detection as a multi-label task of detecting targeted identities using a demographic-focused taxonomy.We then systematically evaluate models across scales and techniques, including prompting, in-context learning, and fine-tuning.Using twelve datasets spanning diverse content types and demographics, our study demonstrates the promise of fine-tuned smaller models for scalable detection.However, our analyses also expose persistent gaps across demographic axes and multi-demographic targeted biases, underscoring the need for more e!ective and scalable detection frameworks.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: hybrid
OpenAlex ID: W7166533082