Multimodal sentiment analysis via dynamic modality routing and cross-modal representation alignment
Year: 2026
Type: article
Abstract: Multimodal sentiment analysis (MSA) aims to estimate the continuous positive–negative affective orientation of an utterance from textual, acoustic, and visual modalities. Despite its potential, practical MSA faces a critical “modality imbalance” challenge: emotional expressions are inherently uneven across modalities and fluctuate dynamically based on the context or speaker. Existing paradigms often struggle with this, as they implicitly assume uniform modality contributions or rely on indirect gating mechanisms that fail to capture sample-level variations in predictive reliability. Furthermore, many fusion strategies overlook the explicit semantic consistency between fused and unimodal features, leading to “semantic drift,” where the integrated representation loses the discriminative strengths of individual modalities. To address these limitations, we propose CAMoE, a multimodal sentiment analysis framework integrating dynamic modality routing and cross-modal representation alignment. CAMoE introduces three synergistic components: (1) a unimodal auxiliary refinement module (UARM), which preserves modality-specific predictive capability through auxiliary unimodal supervision; (2) an adaptive temperature router (ATR), which utilizes a mixture-of-experts (MoE) architecture to dynamically allocate modality weights by adjusting the sharpness of the routing distribution per sample; (3) a cross-modal alignment module (CAM), which enforces reliability-aware semantic consistency. By anchoring fused representations to high-reliability unimodal references, CAMoE effectively preserves individual modality integrity while achieving seamless multimodal integration. Extensive evaluations on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that CAMoE achieves competitive or superior performance compared with the evaluated baseline methods under controlled, feature-matched evaluations. Our results support that explicitly modeling modality reliability provides a more effective and adaptive solution for handling imbalanced data in real-world affective computing systems. Source code is available at https://github.com/TT-TT1/CAMoE .
Authors: Zhiyi Feng, Maoshen Jia
Institution: Beijing University of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,890
OpenAlex ID: W7225638907
FWCI:
Cites:
Cited by:
Related to: