Work

A Graph-enhanced Multimodal Transformer Framework for Continuous Sign Language Recognition and Gloss-assisted Translation

Year: 2026
Type: article
Abstract: Continuous sign language translation (CSLT) aims to convert continuous signing into spoken-language text, but fixed multimodal fusion may fail to adapt to changing visual cues.We propose a graph-enhanced multimodal Transformer using gloss-assisted training with gloss-free inference.The framework combines RGB appearance, skeletal pose, and facial landmarks through adaptive cross-modal weighting and a temporal saliency gate.A CTCbased recognition objective and an autoregressive translation objective are jointly optimized during training, while gloss prediction is not required during inference.On PHOENIX-2014T, the model achieved 17.80 ± 0.20% WER, 25.40 ± 0.12 BLEU-4, and 34.80 ± 0.15 ROUGE-L.On CSL-Daily, it improved BLEU-4 by 0.70 over the strongest baseline.The framework achieved 22 FPS inference throughput under the reported configuration, while ablation and robustness experiments indicated contributions from multimodal fusion and resilience to degraded inputs.
Language: English
Open Access status: hybrid
OpenAlex ID: W7235659839
FWCI:
Cites: 18
Cited by:
Related to: 10