Dynamic gating and attention masking for multimodal conditional diffusion image translation in autonomous driving scenarios
Year: 2026
Type: article
Abstract: In image translation from virtual to real-world autonomous driving scenarios, the conditional diffusion model for multimodal data fusion employs a multi-head self-attention mechanism to model cross-modal global dependencies between semantic segmentation maps and depth maps for scene generation. However, it still has limitations: the translated results exhibit discrepancies in the consistency between semantic contours and spatial depth, failing to meet high-precision requirements; multimodal data quality is uneven, and using a fixed-weight fusion method is susceptible to the influence of low-quality modalities; background noise and modal noise weaken the constraining effect of key features on the denoising process. To address these issues, this paper proposes an improved multi-modal feature fusion framework. The multi-head self-attention mechanism incorporates a dynamic gating module, enabling cross-modal feature weights to be adaptively modulated through spatial semantic importance assessment and channel-modal contribution quantification, thereby balancing the proportions of high- and low-quality modalities. An attention masking mechanism is introduced, using learnable masks to filter out interference from non-critical regions, thereby enhancing the representation of core elements such as vehicles and traffic signs. The optimized multimodal features are incorporated into the denoising process of the diffusion model as dual conditions, guiding the noise prediction network to align semantic and depth constraints at the pixel level, thereby achieving precise translation from virtual to real scenes. Experimental results show that the improved model achieves significantly enhanced translation accuracy on the Cityscapes dataset, with superior performance on metrics such as Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS), achieving values of 42.80 and 0.412, respectively.
Source: Scientific Reports
Authors: Xiaoli Zhang, Mengxiang Liu, Yusi Wang
Institution: Changsha Institute of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7204899163
FWCI:
Cites:
Cited by:
Related to: