A digital-driven deep learning framework for text–image multimodal machine translation with gated cross-modal fusion
Year: 2026
Type: article
Abstract: Text-only neural machine translation (NMT) models can struggle with lexical, gender, and referential ambiguities when textual cues alone are insufficient. This paper presents a digital-driven deep-learning framework for text–image multimodal machine translation that integrates visual context through gated cross-modal fusion and contrastive text-image alignment. The architecture combines a frozen CLIP ViT-B/32 vision encoder with a trainable linear projection and a pretrained MarianMT text encoder-decoder. An adaptive sample-level scalar gate regulates the contribution of visual information according to cross-modal relevance. On the Multi30K EN–DE Test 2016 benchmark, the proposed framework achieves 40.9 BLEU and 0.867 COMET, improving upon text-only MarianMT by 2.2 BLEU and 0.025 COMET and outperforming the evaluated multimodal baselines across seven translation metrics. Paired bootstrap resampling with 1,000 iterations indicates statistically significant improvements ( \(p < 0.001\) , Cohen’s \(d = 0.72\) , 95% CI [1.2, 3.2]). Analysis of gate activations further examines how visual contribution varies across samples with different linguistic characteristics, without assuming token-level gating. The framework introduces an additional inference latency of 21.7 ms over the text-only baseline and is further evaluated under multiple visual degradation and image-relevance conditions. These results demonstrate the potential of adaptive gated fusion for incorporating visual evidence into multimodal machine translation while maintaining computational efficiency.
Source: Scientific Reports
Author: Nan Li
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7226010352
