Emotion Separation and Recognition From a Facial Expression by Generating the Poker Face With Vision Transformers
Year: 2024
Type: article
Abstract: Representation learning and feature disentanglement have garnered significant research interest in the field of facial expression recognition (FER). The inherent ambiguity of emotion labels poses challenges for conventional supervised representation learning methods. Moreover, directly learning the mapping from a facial expression image to an emotion label lacks explicit supervision signals for capturing fine-grained facial features. In this article, we propose a novel FER model, named poker face vision transformer or PF-ViT, to address these challenges. PF-ViT aims to separate and recognize the disturbance-agnostic emotion from a static facial image by generating its corresponding poker face without the need for paired images. Inspired by the facial action coding system, we regard an expressive face as the combined result of a set of facial muscle movements on one's poker face (i.e., an emotionless face). PF-ViT utilizes vanilla vision transformers, and its components are first pretrained as masked autoencoders on a large facial expression dataset without emotion labels, yielding excellent representations. Subsequently, we train PF-ViT using a GAN framework. During training, the auxiliary task of poke face generation promotes the disentanglement between emotional and emotion-irrelevant components, guiding the FER model to holistically capture discriminative facial details. Quantitative and qualitative results demonstrate the effectiveness of our method, surpassing the state-of-the-art methods on four popular FER datasets.
Authors: Jia Li, Jiantao Nie, Dan Guo, Richang Hong, Meng Wang
Institution: Hefei University of Technology
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
Awards: 跨媒体智能问答和推理关键理论与方法研究, G3326758564
OpenAlex ID: W4404056942