Work

When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models

Year: 2026

Type: conference-paper

Abstract: Vision-language models (VLMs) increasingly combine visual and textual information to perform complex tasks.However, conflicts between their internal knowledge and external visual input can lead to hallucinations and unreliable predictions.In this work, we investigate the mechanisms that VLMs use to resolve crossmodal conflicts by introducing WHOOPS-AHA!, a dataset of multimodal counterfactual queries that deliberately contradict internal commonsense knowledge.Through logit inspection, we identify a small set of attention heads that mediate this conflict.By intervening in these heads, we can steer the model towards its internal parametric knowledge or the visual information.Our results show that attention patterns on these heads effectively locate image regions that influence visual overrides, providing a more precise attribution compared to gradientbased methods.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.642

Authors: Francesco Ortu, Zhijing Jin, Diego Doimo, Alberto Cazzaniga

Institutions: University of Trieste, AREA Science Park, University of Toronto, Vector Institute

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: European Commission

OpenAlex ID: W4416834506