Work

Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision–Language Models

Year: 2023

Type: conference-paper

Abstract: Recent breakthroughs in self-supervised training have led to a new class of pretrained vision-language models.While there have been investigations of bias in multimodal models, they have mostly focused on gender and racial bias, giving much less attention to other relevant groups, such as minorities with regard to religion, nationality, sexual orientation, or disabilities.This is mainly due to lack of suitable benchmarks for such groups.We seek to address this gap by providing a visual and textual bias benchmark called MMBias, consisting of around 3,800 images and phrases covering 14 population subgroups.We utilize this dataset to assess bias in several prominent selfsupervised multimodal models, including CLIP, ALBEF, and ViLT.Our results show that these models demonstrate meaningful bias favoring certain groups.Finally, we introduce a debiasing method designed specifically for such large pretrained models that can be applied as a post-processing step to mitigate bias, while preserving the remaining accuracy of the model.

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2023.eacl-main.126

Authors: Sepehr Janghorbani, Gerard De Melo

Institutions: Rutgers, The State University of New Jersey, Hasso Plattner Institute, University of Potsdam

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDG: Reduced inequalities

Open Access status: hybrid

OpenAlex ID: W4386566708