Year: 2026
Type: dissertation
Abstract: Recent advances in large visionlanguage models (LVLMs) have amplified concerns about fairness, yet existing evaluations remain confined to demographic attributes and often conflate fairness with refusal behavior.This thesis broadens the scope of fairness by introducing a counterfactual visual question answering benchmark that probes the decision boundaries of closed-source LVLMs under controlled contextual shifts.Each image pair differs in a single visual attribute that has been validated as irrelevant to the question, enabling ground-truth free and refusal-aware analysis of reasoning stability.Comprehensive experiments reveal that non-demographic attributes, such as environmental context or social behavior, distort LVLM decision-making more strongly than demographic ones.Moreover, instruction-based debiasing shows limited effectiveness and can even amplify these asymmetries, whereas exposure to a small number of human norm validated examples from our benchmark encourages more consistent and balanced responses, highlighting its potential not only as an evaluative framework but also as a means for understanding and improving model behavior.Together, these results provide a practical basis for auditing contextual biases even in black-box LVLMs and contribute to more transparent and equitable multimodal reasoning.i 4.2.1 Main Results . . . . . . . . .
Author: Zhao, Zaiying
Institution: The University of Tokyo
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: green
OpenAlex ID: W7207926915