Work

Deep Mul Timodal Learning for Emotion Recognition in Spoken Language

Year: 2018

Type: conference-paper

Abstract: In this paper, we present a novel deep multimodal framework to predict human emotions based on sentence-level spoken language. Our architecture has two distinctive characteristics. First, it extracts the high-level features from both text and audio via a hybrid deep multimodal structure, which considers the spatial information from text, temporal information from audio, and high-level associations from low-level handcrafted features. Second, we fuse all features by using a three-layer deep neural network to learn the correlations across modalities and train the feature extraction and fusion modules together, allowing optimal global fine-tuning of the entire structure. We evaluated the proposed framework on the IEMOCAP dataset. Our result shows promising performance, achieving 60.4% in weighted accuracy for five emotion categories.

Source: IEEE International Conference on Acoustics Speech and Signal Processing

DOI: 10.1109/icassp.2018.8462440

Authors: Yue Gu, Shuhong Chen, Ivan Marsic

Institution: Rutgers, The State University of New Jersey

Language: English

Topic: Emotion and Mood Recognition

Subfield: Experimental and Cognitive Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: green

Funder: U.S. National Library of Medicine

Award: Intention-aware Recommender System for Improving Trauma Resuscitation Outcomes