Will the Prince Get True Love’s Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts
Year: 2025
Type: conference-paper
Abstract: In this paper, we study whether language models are affected by learned gender stereotypes during the comprehension of stories.Specifically, we investigate how models respond to gender stereotype perturbations through counterfactual data augmentation.Focusing on Question Answering (QA) tasks in fairytales, we modify the FairytaleQA dataset by swapping gendered character information and introducing counterfactual gender stereotypes during training.This allows us to assess model robustness and examine whether learned biases influence story comprehension.Our results show that models exhibit slight performance drops when faced with gender perturbations in the test set, indicating sensitivity to learned stereotypes.However, when finetuned on counterfactual training data, models become more robust to anti-stereotypical narratives.Additionally, we conduct a case study demonstrating how incorporating counterfactual anti-stereotype examples can improve inclusivity in downstream applications.
Authors: Christina A Chance, Da Yin, Dakuo Wang, Kai-Wei Chang
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4411119060