A framework for evaluating cultural bias and historical misconceptions in LLMs outputs
Year: 2025
Type: article
Abstract: Large Language Models (LLMs), while powerful, often perpetuate cultural biases and historical inaccuracies from their training data, marginalizing underrepresented perspectives. To address this, we introduce a structured framework to systematically evaluate and quantify these deficiencies. Our methodology combines culturally sensitive prompting with two novel metrics: the Cultural Bias Score (CBS) and the Historical Misconception Score (HMS). Our analysis reveals varying cultural biases across LLMs, with certain Western-centric models like Gemini exhibiting higher biases. In contrast, other models, including ChatGPT and Poe, demonstrate more balanced cultural narratives. We also find historical misconceptions are most prevalent for less-documented events, underscoring the critical need for training data diversification. Our framework suggests the potential effectiveness of bias mitigation techniques, including dataset augmentation and human-in-the-loop (HITL) verification. Empirical validation of these strategies remains an important direction for future work. This work provides a replicable and scalable methodology for developers and researchers to ensure the responsible and equitable deployment of LLMs in critical domains like education and content moderation.
Authors: Moon-Kuen Mak, Tiejian Luo
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: hybrid
Estimated APC paid:
OpenAlex ID: W4413304752