Work

Using Linguistic Formalism to Improve Real World Understanding for V&L Models: Case Study on Image Discrimination for Structurally Ambiguous Language Input

Year: 2025

Type: article

Abstract: In the context of Real World Understanding (RWU) for vision and language (V&L) models, accurately aligning language with the corresponding visual scene is critical. Since current models typically assume language inputs to be plain text, RWU faces potential issues with structural ambiguity, where a single sentence can have multiple meanings due to various phrase structures. This paper proposes to use linguistic formalism as input, which enriches language information and addresses the issue of structural ambiguity. Our focus is on the Contrastive Language-Image Pre-training (CLIP) model, a prominent V&L model, focusing on image discrimination tasks of RWU. Our experiments test various approaches to incorporating formalism into the CLIP model, depending on the type of formalism and its processing method. We aim to determine the effectiveness of formalism in discriminating ambiguous images and identify which formalism works best. Additionally, we employ a gradient-based method to gain insights into how formalism is interpreted within the model’s architecture.

Source: Journal of Natural Language Processing

DOI: 10.5715/jnlp.32.598

Authors: Lee Sangmyeong, Seitaro Shinagawa, Koichiro Yoshino, Satoshi Nakamura

Institutions: RIKEN, Nara Institute of Science and Technology, Institute of Science Tokyo, Chinese University of Hong Kong, Shenzhen

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funder: Japan Society for the Promotion of Science

Awards: Knowledge Inference System for Robots Bridging Commonsense Written by Language and Real World Observation, Establishment of an Image Generation Platform Based on Structured Representations Linked to Natural Language, Scene Understanding and Knowledge Extension by Active Open-world Recognition