Dual-Graph Multimodal Named Entity Recognition With Optimal Transport Fusion
Year: 2026
Type: article
Abstract: Multimodal Named Entity Recognition (MNER) aims to leverage both textual and visual information to perform entity recognition and classification tasks collaboratively, holding significant application value in scenarios such as social media. However, existing methods predominantly rely on a single visual perspective for cross-modal fusion, struggling to effectively address visual noise issues arising from semantic inconsistencies between images and text. Additionally, deficiencies persist in cross-modal alignment. To address this, this paper proposes a Dual-Graph Multimodal Named Entity Recognition model with Optimal Transport (OT) fusion, denoted as DG-OT-MNER. The model constructs two structurally consistent cross-modal graphs—one based on the original image and another generated from text-guided enhanced images—enabling multimodal information propagation through a Graph Attention Network (GAT). It further introduces an optimal transport mechanism to perform fine-grained alignment and dynamic fusion of text representations across different visual perspectives. This design reduces noise interference from visual data while enhancing multimodal semantic consistency, thereby improving the model’s entity recognition and disambiguation capabilities. Extensive experiments on the Twitter-2015 and Twitter-2017 datasets demonstrate that the proposed method achieves significant improvements in overall performance and across multiple entity categories, with Overall F1 scores reaching 75.65% and 87.20% respectively, outperforming existing mainstream approaches. Further ablation studies and case analyses validate the effectiveness and necessity of the dual-graph structure and optimal transport fusion strategy for multimodal named entity recognition tasks.
Source: IEEE Access
Authors: Shihong Wu, Yupeng Ma, Ming Zhang
Institution: Wuhan Institute of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W7134831266
FWCI:
Cites:
Cited by:
Related to: