Many Cultures, One Label: Investigating Cultural Alignment in Harmful Meme Detection
Year: 2026
Type: conference-paper
Abstract: Harmful meme detection has become an important multimodal classification task. However, existing approaches are trained and evaluated against a single ground-truth binary label (harmful/not harmful), implicitly assuming a universal notion of harmfulness despite evidence that perceptions of harmful content are shaped by cultural and social context. In this paper, we investigate whether culture-aligned Large Language Model (LLM) agents can surface cross-cultural variation in harmful meme perception and reveal potential biases in benchmark annotations. To this end, we instantiate six culture-aligned LLM agents. Given a meme, each agent independently evaluates its harmfulness and provides a culturally grounded rationale. These assessments are integrated with OCR text and image captions into a concept graph representation of the meme, and combined with complementary vision language features to train a downstream classifier. Experiments on the Facebook Hateful Memes (FHM) benchmark show that the resulting model achieves competitive performance (AUC = 0.804), outperforming several baselines. More importantly, our analysis reveals substantial cross-cultural variation: the six agents disagree on 34.9% of 10,000 evaluated memes, while maintaining an overall agreement level (k = 0.690) remarkably close to that reported for human annotators (k = 0.684). We further find significant differences in alignment with the ground-truth labels, with agents representing Brazil, South Africa, and the United States exhibiting consistently higher agreement than agents representing India, Japan, and Saudi Arabia. These findings suggest that harmfulness benchmarks may implicitly privilege particular cultural interpretations of harmful content.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7225117983