Cross-modal BERT model for enhanced multimodal sentiment analysis in psychological social networks
Year: 2025
Type: article
Abstract: BACKGROUND: Human emotions in psychological social networks often involve complex interactions across multiple modalities. Information derived from various channels can synergistically complement one another, leading to a more nuanced depiction of an individual's emotional landscape. Multimodal sentiment analysis emerges as a potent tool to process this diverse array of content, facilitating efficient amalgamation of emotions and quantification of emotional intensity. METHODS: This paper proposes a cross-modal BERT model and a cross-modal psychological-emotional fusion (CPEF) model for sentiment analysis, integrating visual, audio, and textual modalities. The model initially processes images and audio through dedicated sub-networks for feature extraction and reduction. These features are then passed through the Masked Multimodal Attention (MMA) module, which amalgamates image and audio features via self-attention, yielding a bimodal attention matrix. Subsequently, textual information is fed into the MMA module, undergoing feature extraction through a pre-trained BERT model. The textual information is then fused with the bimodal attention matrix via the pre-trained BERT model, facilitating emotional fusion across modalities. RESULTS: The experimental results on the CMU-MOSEI dataset showcase the effectiveness of the proposed CPEF model, outperforming comparative models, achieving an impressive accuracy rate of 83.9% and F1 Score of 84.1%, notably improving the quantification of negative, neutral, and positive affective energy. CONCLUSIONS: Such advancements contribute to the precise detection of mental health status and the cultivation of a positive and sustainable social network environment.
Source: BMC Psychology
Author: Jian Feng
Institution: Pingdingshan University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,990
OpenAlex ID: W4414601178