Get started
Work

Dual-Counterfactual Evidence Decomposition and Selective Editing for Hallucination Mitigation in Large Vision-Language Models

Year: 2026
Type: preprint
Abstract: Large vision-language models (LVLMs) often generate fluent but visually unsupported content. Existing training-free methods mainly recalibrate output distributions or edit representations without identifying whether the evidence behind the current claim is visual, linguistic, or shared. This limits intervention precision: unsupported language priors may survive, whereas grounded visual information may be weakened. We introduce a dual-counterfactual evidence decomposition and selective editing framework. Paired visual and linguistic counterfactuals act as sample-specific evidence probes, whose hidden-state responses are decomposed into shared, visual-exclusive, and linguistic-exclusive components. A claim-aware gate attenuates unsupported linguistic evidence only for visually verifiable claims, while an evidence-preserving update protects shared and visual-exclusive support. The method is training-free and requires neither parameter updates nor external verification models. Across three LVLM families, it achieves the highest Accuracy and F1 in all nine POPE backbone--split settings. On CHAIR, it obtains the lowest CHAIR_i on all three evaluated backbones, reducing it by 9.1%-14.7% relative to HulluEdit while maintaining comparable BLEU scores. It also achieves the highest aggregate MME-H on all three backbones, the best results on all four HallusionBench metrics, and a score of 29.3 on MM-Vet. At 23.59 tokens per second, the complete pipeline retains 75.5% of greedy-decoding throughput. These results establish evidence attribution before intervention as an effective and practical strategy for faithful LVLM generation.
DOI: 10.21203/rs.3.rs-11118276/v1
Institution: Ningxia University
Language: English
Open Access status: green
OpenAlex ID: W7225051005
Cites: 18
Cited by:
Related to: 10