Work

Multimodal Emotion Recognition: Integrating Audio and Visual Features Using Enhanced Deep Learning Techniques

Year: 2024
Type: article
Abstract: Emotion recognition is a pivotal area in human-computer interaction, crucial for enhancing system responsiveness and adaptability. The expression of human emotion depends on various verbal and non-verbal. Emotion recognition is thus well suited as a multimodal rather than single-modal learning problem. This study introduces Multimodal that integrates speech (Audio) and facial features to recognize three primary emotions: happiness, sadness, and surprise from a video dataset (MELD). In audio feature extraction, an autoencoder is used, which improves the model's capacity to identify subtle emotional subtleties from speech signals. Concurrently, ResNet is used to extract image features by transfer learning, using pre trained weights to identify intricate visual patterns from summary pictures. The Improved Zebra Algorithm (IZA) is used in feature selection to maximize discriminative feature subsets. Our suggested Bi- Directional LSTM with self-attention mechanism is evaluated by comparison with two baseline models, namely Bi Directional LSTM and Convolutional Neural Network (CNN). Our method achieves state-of-art results on MELD. More specifically, the highest accuracy was obtained by the Bi-LSTM-self attention model with 89.83%, followed by 85.15% by the Bi-LSTM, and 86.87% by the CNN respectively. These findings demonstrate the efficiency of the Bi-LSTM- SA model on multimodal emotion recognition.
Author: Archna Kirar
Language: English
Open Access status: diamond
OpenAlex ID: W7216709960
FWCI:
Cites: 33
Cited by:
Related to: 10