Work

Joint Fine-Grained Entity-Type Disambiguation and Robust Grounding via Multi-Level Knowledge Integration for FMNERG

Year: 2026
Type: article
Abstract: Fine-grained Multimodal Named-Entity Recognition and Grounding (FMNERG) aims to extract entity–type–object triples from image–text pairs. However, existing methods still struggle with two major challenges: distinguishing semantically similar labels in an expanded fine-grained label space and achieving robust entity–object alignment under complex image–text relations in social media. To address these issues, we propose a Multi-level Knowledge Integration Framework (MKIF) for FMNERG, which leverages multi-level knowledge to jointly improve fine-grained entity-type disambiguation and robust entity grounding in a unified manner. Specifically, we employ multimodal large language models (MLLMs) to obtain global-level knowledge, enriching the semantics of raw multimodal inputs and serving as a shared semantic anchor across subtasks. To improve fine-grained entity-type discrimination, we further derive conceptual and discriminative descriptions for each type as label-level knowledge and utilize both global- and label-level knowledge as supervision via contrastive learning. In addition, we discover potential visual concepts in the dataset as object-level knowledge and design a dynamic gating mechanism to incorporate the global- and object-level knowledge into object embeddings for robust entity–object alignment. Comprehensive experiments demonstrate that MKIF outperforms state-of-the-art methods. Furthermore, detailed analyses validate the effectiveness of each core component.
Source: Electronics
Language: English
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7225636274
FWCI:
Cites: 53
Cited by:
Related to: 10