Work

Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

Year: 2020

Type: conference-paper

Abstract: In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts.Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context.To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations.To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions.Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2020.acl-main.306

Authors: Jianfei Yu, Jing Jiang, Li Yang, Rui Xia

Institutions: Nanjing University of Science and Technology, Singapore Management University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Research Foundation, National Research Foundation Singapore, National Natural Science Foundation of China

Award: G3530307534