Work

Debiasing Pre-Trained Language Models via Efficient Fine-Tuning

Year: 2022

Type: conference-paper

Abstract: An explosion in the popularity of transformerbased language models (such as GPT-3, BERT, RoBERTa, and ALBERT) has opened the doors to new machine learning applications involving language modeling, text generation, and more.However, recent scrutiny reveals that these language models contain inherent biases towards certain demographics reflected in their training data.While research has tried mitigating this problem, existing approaches either fail to remove the bias completely, degrade performance ("catastrophic forgetting"), or are costly to execute.This work examines how to reduce gender bias in a GPT-2 language model by fine-tuning less than 1% of its parameters.Through quantitative benchmarks, we show that this is a viable way to reduce prejudice in pre-trained language models while remaining cost-effective at scale.

Source: Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI)

DOI: 10.18653/v1/2022.ltedi-1.8

Authors: Michael Gira, Ruisu Zhang, Kangwook Lee

Institution: University of Wisconsin–Madison

Language: English

Topic: Ethics and Social Impacts of AI

Subfield: Safety Research

Field: Social Sciences

Domain: Social Sciences

SDGs: Reduced inequalities, Gender equality

Open Access status: hybrid

Funders: National Science Foundation, Wisconsin Alumni Research Foundation, University of Wisconsin-Madison, Division of Graduate Education