MSTrack: Multi-Modal Feature Aggregation and Spatio-Temporal Memory Context Encoding for Ground and UAV RGB-Thermal Infrared Tracking
Year: 2026
Type: article
Abstract: RGB-thermal infrared (RGBT) target tracking integrates spatial features of both RGB and thermal infrared (TIR) modalities, providing a discriminative basis for target feature representation and state estimation. However, existing methods suffer from insufficient cross-modal feature interaction and inadequate spatio-temporal contextual correlation, which limits tracking stability and accuracy in challenging scenarios. To address these issues, we propose MSTrack, a novel tracker that integrates a multi-modal feature aggregation (MMFA) module and a spatio-temporal memory context encoding (STMCE) module. First, to capture both shared features and complementary details across modalities, we design the MMFA module, which employs parallel Swin Transformers and four cross-modal feature interaction (CMFI) components to achieve progressive fusion of RGB and TIR features. Specifically, CMFI integrates a cross-modal collaborative attention mechanism, a triplet attention mechanism, a bidirectional cross-modal weighted fusion mechanism, and a multi-stage information aggregation mechanism to achieve cross-modal interactions. Second, to fuse the key spatio-temporal features across multiple frames, we design the STMCE module. This STMCE module leverages a historical feature memory pool (HFMP) and a spatio-temporal context encoder (STCE) to model global dependencies among the template, current, and historical frames. Finally, the prediction head outputs the target bounding box. Experiments on four ground-based RGBT datasets demonstrate that the proposed tracker outperforms other state-of-the-art trackers. Furthermore, the results on the UAV-based VTUAV dataset validate its generalization potential. Specifically, MSTrack achieves PR/SR scores of 0.877/0.637 on RGBT234 and 0.669/0.530 on LasHeR. On the VTUAV dataset, it achieves MPR/MSR scores of 0.812/0.672 on the short-term subset and 0.595/0.501 on the long-term subset.
Source: Remote Sensing
DOI: 10.3390/rs18224004
Institutions: Xidian University, Kyungpook National University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,258
Awards: G437369669, G5869524761, G6602798985
OpenAlex ID: W7237219329
