GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution
Year: 2022
Type: other
Abstract: Multimodal Coreference Resolution (MCR) involves resolving the references across different modalities, which is a crucial capability for building next-generation conversational agents. MCR is challenging as it requires encoding information from different modalities and modeling associations between them. Though significant progress has been made for visual-linguistic tasks such as visual grounding, most of the current works involve single turn utterances and focus on simple coreference resolutions. In this work, we propose an MCR model that resolves coreference made in multi-turn dialogues with scene images. We present GraVL-BERT, a unified MCR framework which combines visual relationships across objects, background scenes, dialogue, and metadata by integrating Graph Neural Networks with VL-BERT. We present results on the SIMMC 2.0 multimodal conversational dataset, achieving the rank-1 on the DSTC-10 SIMMC 2.0 MCR challenge with F1 score 0.783.
Source: Underline Science Inc.
DOI: 10.48448/phm0-3037
Authors: International Committee for Computational Linguistics 2022, Agarwal, Sanchit, Bansal, Mohit, Biswas, Arijit, Chung, Tagyoung, Gao, Bill, GUO, DANFENG, Gupta, Arpit, Kao, Jiun-Yu, Lin, Chien-Wei
Institutions: Amazon (United States), University of California, Los Angeles
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6891920313
Cites:
Cited by:
Related to: