Work

FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models

Year: 2026

Type: conference-paper

Abstract: Vision-language models (VLMs) are increasingly deployed in multi-turn settings where users may describe visual content with incorrect assumptions.Yet existing evaluations rarely isolate how models respond when the same visually grounded false premise persists across dialogue turns.We introduce FPCO-Dialog, a benchmark for evaluating correction and cooperation behavior in VLMs under repeated false premises.FPCO-Dialog contains 1,080 images and 10,800 question turns, stratified by visual complexity, object category, and false-premise class, and uses a 10-turn protocol in which a correct dialogue prefix is followed by repeated false-premise referring expressions.We evaluate 20 commercial and open-source VLMs with a modelagnostic protocol and CorrTP@K, a correctionrate metric over false-premise turns, scored by two independent detectors.FPCO-Dialog reveals substantial and persistent cross-model differences in aggregate correction tendency, model-specific turn-wise dynamics, and systematic variation across false-premise types under the benchmark's substitution distribution.The dataset, evaluation protocol, model outputs, detector labels, and code are available at https: //github.com/lab-klc/FPCO-Dialog. Corresponding author.Does the blue and yellow bus on the tracks appear to be at a station platform?Yes, it appears to be at a station platform.There is no bus in the image.The vehicle on the tracks is a train.However, to answer the rest...No, it takes up only a moderate portion of image.Just to clarify, it's actually a train, not a bus.But yes, it does take up a large portion of the image... Cooperation Cooperation Correction CorrectionDoes the blue and yellow bus on the tracks take up a large part of the image?

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/2026.emnlp-main.286