Work

Taming Object Hallucinations with Verified Atomic Confidence Estimation

Year: 2026

Type: conference-paper

Abstract: Multimodal Large Language Models (MLLMs) often suffer from hallucinations, particularly errors in object existence, attributes, or relations, which undermine their reliability.We introduce TACO (Verified Atomic Confidence Estimation), a simple framework that mitigates hallucinations through self-verification and confidence calibration without relying on external vision experts.TACO decomposes responses into atomic queries, paraphrases them to reduce sensitivity to wording, and estimates confidence using selfconsistency (black-box) or self-confidence (gray-box) aggregation, before refining answers with a language model.Experiments on five benchmarks (POPE, MME, Hallusion-Bench, AMBER, and MM-Hal Bench) with two MLLMs (LLaVA-1.5-7Band CogVLM2) show that TACO consistently outperforms direct prompting and Visual Contrastive Decoding, reduces systematic biases, and improves confidence calibration, demonstrating its effectiveness in enhancing the faithfulness of MLLMs.* attribute -shape * attribute -size * attribute -count * attribute -texture * attribute -style * attribute -temporal

Source: Conference of the European Chapter of the Association for Computational Linguistics

DOI: 10.18653/v1/2026.eacl-long.252

Authors: Jiarui Liu, Weihao Xuan, Zhijing Jin, Mona T. Diab

Institutions: Carnegie Mellon University, RIKEN, RIKEN Center for Advanced Intelligence Project, The University of Tokyo, University of Toronto

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Canadian Institute for Advanced Research, Government of Canada, Bundesministerium für Bildung und Forschung

Awards: Verbundprojekt TUEAI: Tübingen AI Center, G4980694453