Work

Fusical: Multimodal Fusion for Video Sentiment

Year: 2020
Type: article
Abstract: Determining the emotional sentiment of a video remains a challenging task that requires multimodal, contextual understanding of a situation.In this paper, we describe our entry into the EmotiW 2020 Audio-Video Group Emotion Recognition Challenge to classify group videos containing large variations in language, people, and environment, into one of three sentiment classes.Our end-toend approach consists of independently training models for different modalities, including full-frame video scenes, human body keypoints, embeddings extracted from audio clips, and image-caption word embeddings.Novel combinations of modalities, such as laughter and image-captioning, and transfer learning are further developed.We use fully-connected (FC) fusion ensembling to aggregate the modalities, achieving a best test accuracy of 63.9% which is 16 percentage points higher than that of the baseline ensemble.
Language: English
Open Access status: green
OpenAlex ID: W4289213428
FWCI:
Cites: 26
Cited by:
Related to: 10