Self-Supervised Masking for Interpretable and Robust Facial Emotion Recognition
Year: 2026
Type: dissertation
Abstract: Facial emotion recognition (FER) is a critical component of human-computer interaction, with applications in mental health assessment, driver safety, and security systems. However, real-world FER systems face two persistent challenges: severe class imbalance that causes models to fail on rare emotions such as disgust and fear, and a lack of interpretability in black-box deep learning models. This thesis addresses both challenges by proposing Mask-CAM (Masked Class Activation Mapping), a lightweight and intrinsically interpretable convolutional neural network that learns a self-supervised soft mask to suppress background regions and focus on discriminative facial features without requiring external mask labels. The proposed architecture extends a standard CAM network with a mask generator driven by competing losses: cross-entropy for emotion recognition, a sparsity penalty that encourages the mask to suppress as many regions as possible, and a smoothness (total variation) penalty that ensures contiguous, blob-like masks. A warm-up schedule gradually introduces the sparsity pressure during training. Experiments on FERPlus (in-distribution) and RAF-DB (cross-dataset) demonstrate that Mask-CAM achieves competitive accuracy (78.85% on FERPlus, 67.50% on RAF-DB) while attaining the highest macro F1 on FERPlus (65.60%) and strong macro F1 on RAF-DB (46.71%). Crucially, Mask-CAM is the only model to detect disgust on RAF-DB (F1 = 0.05) and improves disgust F1 on FERPlus from 0.00 to 0.38. Overlay accuracy using facial landmarks confirms that the mask focuses on facial regions in 99.3% of FERPlus and 100% of RAF-DB images, and the mask density adapts to emotion difficulty—sparser for happy, denser for disgust. With just 0.48 million parameters, which is six times fewer than a comparable black-box baseline, Mask-CAM is small enough to run in real time on resource-constrained devices. Test-time augmentation (TTA) further boosts accuracy to 80.05% (FERPlus) and 68.16% (RAF-DB), with the smallest cross-dataset drop among all compared models. McNemar's test shows that Mask-CAM's accuracy is not significantly different from the baselines (p > 0.05), confirming that interpretability is added without sacrificing performance. This work demonstrates that a self-supervised soft mask can simultaneously provide built-in interpretability and improved recognition of under-represented emotions.
Author: Lama, Bhim
Institution: Tribhuvan University
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
SDG: Reduced inequalities
Open Access status: green
OpenAlex ID: W7219652870