Work

FOCUS: Evaluating Pre-trained Vision-Language Models on Underspecification Reasoning

Year: 2025

Type: conference-paper

Abstract: Humans possess a remarkable ability to interpret underspecified ambiguous statements by inferring their meanings from contexts such as visual inputs.This ability, however, may not be as developed in recent pre-trained visionlanguage models (VLMs).In this paper, we introduce a novel probing dataset called FO-CUS to evaluate whether state-of-the-art VLMs have this ability.FOCUS consists of underspecified sentences paired with image contexts and carefully designed probing questions.Our experiments reveal that VLMs still fall short in handling underspecification even when visual inputs that can help resolve the ambiguities are available.To further support research in underspecification, FOCUS will be released for public use.We hope this dataset will inspire further research on the reasoning and contextual understanding capabilities of VLMs.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-long.1337

Authors: Kankan Zhou, Eason Lai, Kyriakos Mouratidis, Jing Jiang

Institutions: Singapore Management University, Australian National University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: Ministry of Education - Singapore, Ministry of Education, India

Award: G6854926366