Year: 2026
Type: conference-paper
Abstract: We introduce DualFact, a dual-layer, multimodal factuality evaluation framework for procedural video captioning.DualFact separates factual correctness into conceptual facts, capturing abstract semantic roles (e.g., Action, Ingredient, Tool, Location), and contextual facts, capturing their grounded predicate-argument realizations in video.To support complete and role-consistent evaluation, DualFact incorporates implicit argument augmentation (VIA) and contrastive fact sets.We instantiate Dual-Fact in two modes: DualFact-T, which verifies facts against textual evidence, and DualFact-V, which verifies facts against video-grounded visual evidence.Experiments on YouCook3-Fact and CraftBench-Fact show that state-ofthe-art multimodal language models produce fluent but often factually incomplete captions, with systematic omissions and role-level inconsistencies.DualFact correlates more strongly with human factuality judgments than standard metrics, particularly for contextual facts, and reveals that caption-only evaluation overestimates hallucinations compared to videogrounded verification.Overall, DualFact offers an interpretable and human-aligned evaluation protocol that highlights persistent challenges in multimodal factual grounding, extending beyond surface-level fluency.
Institution: German Research Centre for Artificial Intelligence
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: European Commission
OpenAlex ID: W7166859371