Work

ChatterBox: Multimodal Referring and Grounding with Chain-of-Questions

Year: 2025

Type: conference-paper

Abstract: In this study, we establish a benchmark and a baseline approach for Multimodal referring and grounding with Chain-of-Questions (MCQ), opening up a promising direction for ‘logical’ multimodal dialogues. The newly collected dataset, named CB-300K, spans challenges including probing dialogues with spatial relationship among multiple objects, consistent reasoning, and complex question chains. The baseline approach, termed ChatterBox, involves a modularized design and a referent feedback mechanism to ensure logical coherence in continuous referring and grounding tasks. This design reduces the risk of referential confusion, simplifies the training process, and presents validity in retaining the language model’s generation ability. Experiments show that ChatterBox demonstrates superiority in MCQ both quantitatively and qualitatively, paving a new path towards multimodal dialogue scenarios with logical interactions.

Source: Proceedings of the AAAI Conference on Artificial Intelligence

DOI: 10.1609/aaai.v39i7.32796

Authors: Yunjie Tian, Tianren Ma, Lingxi Xie, Qixiang Ye

Institution: University of Chinese Academy of Sciences

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funder: National Natural Science Foundation of China

Awards: G6735075065, G6906876362