A systematic review of vision transformer and explainable AI advances in multimodal facial expression recognition
Year: 2025
Type: review
Study design: Systematic Review
Abstract: Facial expression is one of the most important indicators used to convey human emotions. Facial expression recognition is the process of automatically detecting and classifying these expressions by computer systems. Multimodal facial expression recognition aims to perform a more accurate and comprehensive emotion analysis by combining facial expressions with different modalities such as image, speech, Electroencephalogram (EEG), or text. This study systematically reviews research conducted between 2021 and 2025 on the Vision Transformer (ViT) based approaches and Explainable Artificial Intelligence (XAI) techniques in multimodal facial expression recognition, as well as the datasets employed in these studies. The findings indicate that ViT-based models outperform conventional Convolutional Neural Networks (CNNs) by effectively capturing long-range dependencies between spatially distant facial regions, thereby enhancing emotion classification accuracy. However, significant challenges remain, including data privacy risks arising from the collection of multimodal biometric information, data imbalance and inter-modality incompatibility, high computational costs hindering real-time applications, and limited progress in model explainability. Overall, this study highlights that integrating advanced ViT architectures with robust XAI and privacy-preserving techniques can enhance the reliability, transparency, and ethical deployment of multimodal facial expression recognition systems. • Survey of multimodal facial expression recognition with ViT and XAI (2021–2025). • Reviews advances using image, speech, EEG, and text modalities. • Analyzes Vision Transformer approaches for emotion understanding. • Explores XAI techniques in multimodal facial expression recognition. • Identifies open challenges and future research directions in multimodal FER.
Authors: Ilya Kus, Cemal Kocak, Ayse Keles
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,790
OpenAlex ID: W4417079826