Work

Three stream fusion network with color aware transformer for image to point cloud registration

Year: 2025

Type: article

Abstract: Driven by the advancements in artificial intelligence technologies, image-to-point-cloud registration (I2P) have made significant strides. Nevertheless, the dimensional differences in the features of points cloud and image continue to pose considerable challenges to their development. The primary challenge is the inability to leverage the features of one modality to augment those of another, which complicates latent space feature alignment. To address this challenge, we propose an I2P method named TFCT-I2P. Initially, we introduce a Three-Stream Fusion Network (TFN), which integrates color information from images with structural information from point clouds, facilitating the alignment of features from both modalities. Subsequently, to effectively mitigate patch-level misalignments introduced by the inclusion of color information, we design a Color-Aware Transformer (CAT). Finally, we conduct extensive experiments on 7Scenes, RGB-D Scenes V2, ScanNet V2, and a self-collected dataset. The results demonstrate that TFCT-I2P surpasses state-of-the-art methods. Therefore, we believe that the proposed TFCT-I2P contributes to the advancement of I2P registration. The source code will be released at https://github.com/muyao99/TFCT-I2P soon.

Source: Scientific Reports

DOI: 10.1038/s41598-025-28729-5

Authors: Muyao Peng, Pei An, Zichen Wan, You Yang, Qiong Liu

Institution: Huazhong University of Science and Technology

Language: English

Topic: 3D Shape Modeling and Analysis

Subfield: Computational Mechanics

Field: Engineering

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,690

Funders: National Natural Science Foundation of China, China Postdoctoral Science Foundation, National Key Research and Development Program of China