Work

Multimodal Sentiment-Symptom Disentanglement network for interpretable mental health classification from text and visual data

Year: 2026
Type: article
Abstract: Mental health disorders manifest through intertwined affective and symptomatic signals that are difficult to separate using unimodal or naively fused multimodal models, often producing classifiers that conflate general sentiment polarity with clinically meaningful symptom markers and offer little insight into their own decision process. This paper proposes a Multimodal Sentiment-Symptom Disentanglement Network (MSSD-Net) that explicitly decomposes textual and visual representations into two orthogonal latent subspaces — a sentiment-affect subspace capturing general emotional tone and a symptom-specific subspace capturing clinically relevant behavioral and linguistic markers (e.g., psychomotor cues, facial action units, negation and rumination patterns in text). Modality-specific transformer encoders first extract joint text and visual embeddings, which are passed through a disentanglement module trained with a combination of mutual-information minimization, orthogonality regularization, and adversarial subspace classifiers to enforce separation between the two latent streams. A cross-modal attention fusion layer then recombines the disentangled symptom subspaces from both modalities for downstream mental health classification (e.g., depression, anxiety severity levels). At the same time, the sentiment subspace is retained only as an auxiliary, interpretability-supporting signal. To ensure clinical transparency, the framework incorporates a post-hoc and intrinsic explainability layer that projects attention weights and subspace activations back onto input tokens and facial regions, yielding symptom-attributed saliency maps rather than sentiment-confounded heatmaps. Experiments on benchmark multimodal mental health datasets demonstrate that MSSD-Net achieves 89.4% F1-score over E-DAIC and state-of-the-art multimodal baselines, while substantially improving the faithfulness and clinician-rated interpretability of the generated explanations. Ablation studies reveal that disentangling sentiment and symptom is responsible for the performance gain and improved interpretability in joint entangled fusion. The findings indicate that disentangled multimodal architectures can serve as promising, clinically sound methods for automated mental health screening from naturalistic, real-world data.
Institution: SR University
Language: English
Field: Psychology
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7234535176
FWCI:
Cites: 19
Cited by:
Related to: 10