Year: 2023
Type: preprint
Abstract: Generative models (GMs) have received increasing research interest for their remarkable capacity to achieve comprehensive understanding. However, their potential application in the domain of multi-modal tracking has remained relatively unexplored. In this context, we seek to uncover the potential of harnessing generative techniques to address the critical challenge, information fusion, in multi-modal tracking. In this paper, we delve into two prominent GM techniques, namely, Conditional Generative Adversarial Networks (CGANs) and Diffusion Models (DMs). Different from the standard fusion process where the features from each modality are directly fed into the fusion block, we condition these multi-modal features with random noise in the GM framework, effectively transforming the original training samples into harder instances. This design excels at extracting discriminative clues from the features, enhancing the ultimate tracking performance. To quantitatively gauge the effectiveness of our approach, we conduct extensive experiments across two multi-modal tracking tasks, three baseline methods, and three challenging benchmarks. The experimental results demonstrate that the proposed generative-based fusion mechanism achieves state-of-the-art performance, setting new records on LasHeR and RGBD1K.
Source: arXiv (Cornell University)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Natural Science Foundation of China, Ministry of Education of the People's Republic of China, Higher Education Discipline Innovation Project, Defence Science and Technology Laboratory, Multidisciplinary University Research Initiative, Engineering and Physical Sciences Research Council
Awards: (N00014-16-R-FO05) Semantic Information Pursuit for Multimodal Data Analysis, G1106734755, G1474490597, Face Matching for Automatic Identity Retrieval, Recognition, Verification and Management, G283044312, G334333299, G344950217, G5081062661, G575726069, G5773036057, G5818953154, 稀疏低秩表示优化理论与新方法研究, G8403763664
OpenAlex ID: W4386528663