Work

Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

Year: 2022

Type: article

Abstract: This paper describes a new posed multimodal emotional dataset and compares human emotion classification based on four different modalities - audio, video, electromyography (EMG), and electroencephalography (EEG). The results are reported with several baseline approaches using various feature extraction techniques and machine-learning algorithms. First, we collected a dataset from 11 human subjects expressing six basic emotions and one neutral emotion. We then extracted features from each modality using principal component analysis, autoencoder, convolution network, and mel-frequency cepstral coefficient (MFCC), some unique to individual modalities. A number of baseline models have been applied to compare the classification performance in emotion recognition, including k-nearest neighbors (KNN), support vector machines (SVM), random forest, multilayer perceptron (MLP), long short-term memory (LSTM) model, and convolutional neural network (CNN). Our results show that bootstrapping the biosensor signals (i.e., EMG and EEG) can greatly increase emotion classification performance by reducing noise. In contrast, the best classification results were obtained by a traditional KNN, whereas audio and image sequences of human emotions could be better classified using LSTM.

Source: IEEE Access

DOI: 10.1109/access.2022.3146729

Authors: Jin Chen, Tony Ro, Zhigang Zhu

Institutions: City College of New York, City University of New York, The Graduate Center, CUNY

Language: English

Topic: Emotion and Mood Recognition

Subfield: Experimental and Cognitive Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: gold

Estimated APC paid: $1,850