Work

Text-Assisted Multimodal Adaptive Registration and Fusion Classification Network

Year: 2025

Type: article

Abstract: Due to inherent sensor discrepancies, hyperspectral image (HSI) and LiDAR data in fusion classification tasks often suffer from spatial misalignment and limited high-quality annotations, which severely constrain classification performance. While some recent methods attempt to address these issues by introducing semantic alignment strategies or contrastive learning (CL), challenges such as inconsistent cross-modal representations and limited semantic generalization still persist. To address these issues, we propose a novel text-assisted adaptive registration and fusion classification network (TARCNet). First, we develop a feature adaptive alignment (FAA) module, which adaptively adjusts LiDAR features to alleviate semantic inconsistency under misregistration. Second, we introduce a text-assisted contrastive learning (TCL) module, which leverages linguistic priors to strengthen cross-modal consistency and improve the discriminability of the learned representations. Third, we incorporate a multi-loss joint optimization (MLO) module to ensure consistent and stable optimization across heterogeneous modalities. Extensive experiments conducted on three HSI-LiDAR datasets with misregistration and limited annotations demonstrate that our method outperforms several state-of-the-art approaches, validating its effectiveness and generalization capability.

Source: IEEE Transactions on Geoscience and Remote Sensing

DOI: 10.1109/tgrs.2025.3649754