Work

Entity-level cross-modal fusion for multimodal chinese agricultural diseases and pests named entity recognition

Year: 2025

Type: article

Abstract: Named Entity Recognition (NER), as one of the popular directions in natural language processing, plays a critical role in fields such as information extraction and agricultural knowledge graph construction. However, traditional pure text-based methods often face limitations in agricultural entity recognition, including information singularity, contextual constraints, and insufficient feature fusion capabilities. Inspired by multimodal learning, this paper proposes an agricultural NER model based on entity-level cross-modal alignment, combining dual-stream entity-level feature encoding and dynamic gated attention mechanisms. First, we present a Dual-Stream Entity-Level Feature Encoding architecture. The text stream employs a Boundary-Middle (B-M) classification strategy to achieve fine-grained semantic unit segmentation, effectively addressing long-entity boundary ambiguity and parallel computing challenges. The visual stream utilizes saliency region detection to enhance multi-scale visual entity feature extraction capabilities. Second, we propose a Dynamic Cross-modal Gated Attention (DCGA) mechanism that adaptively adjusts visual feature contributions through gating weights, combined with cross-modal contrastive learning to strengthen entity-level semantic associations between text and images. To validate the model's effectiveness, we constructed a multimodal NER dataset containing 12,074 sample pairs across 10 entity categories, covering 10 crops, 82 typical diseases/pests, and related agrochemical data. The proposed method achieves a macro-average F1 score of 90.73% across 10 agricultural entity types, outperforming single-modal baselines by 5.96%, mainstream multimodal NER models by +3.06%, zero-shot GPT models by +11.41%, and fine-tuned multimodal large models by +2.1%. Comprehensive experimental results indicated that our multimodal collaborative learning framework could effectively enhance agricultural entity recognition accuracy, providing reliable technical support for downstream applications such as agricultural knowledge graph construction and intelligent question answering.

Source: Smart Agricultural Technology

DOI: 10.1016/j.atech.2025.101188