Work

Deep hierarchical weighted late fusion model for multimodal emotion recognition

Year: 2024
Type: dissertation
Abstract: Emotion recognition gains its relevance in diverse areas such as healthcare, education, business marketing and entertainment. As time goes on, research in emotion recognition analysis has developed from the traditional single modal to complex multimodal analysis. Extraction of data from various modalities such as audio, text and video are required in order to gain a meaningful pattern, thus increasing the accuracy of the recognition. In this study, the focus was given on implementing the late fusion approach to integrate three modalities for multimodal emotion recognition. Nevertheless, the late fusion approach faces several challenges, including the loss of fine-grained temporal and spatial information, class imbalance across modalities and reliance on individual model training performance. To address these issues, this study presents the Deep Hierarchical Weighted Late Fusion (DHWLF) model, which is designed for an accurate multimodal emotion recognition. DHWLF employs deep hierarchical weighted late fusion method to effectively reduce the loss of fine-grained temporal and spatial information. This study also introduces individual emotion recognition models for audio, text, and video image modalities using deep learning techniques. Specifically, for the audio, this study introduces the SMOTE-2DCNN model, which combines the Synthetic Minority Oversampling Technique (SMOTE) method with a 2-Dimensional Convolutional Neural Network (2DCNN) to handle imbalanced class distribution and classify emotions accurately. For the text, this study implements the Bidirectional Encoder Representations from Transformers (BERT) language model known as TextBERT model to capture semantic and contextual information from textual data. Lastly, for the video, this study proposes the Fer2DCNN model, which utilizes the Cascade Haar Classifier to detect facial regions before feeding them into a 2DCNN for emotion classification. To evaluate the effectiveness of the DHWLF model, extensive experiments were conducted. The results demonstrated its superiority with the DHWLF model achieving 90% accuracy, outperforming both individual emotion recognition models and other state-of-the-art multimodal emotion recognition models from prior studies.
Language: English
Open Access status: green
OpenAlex ID: W7203567199
Cites: 145
Cited by:
Related to: 9