Who, Where, How Matters: Evaluating Emotion Recognition under Context Shifts
Year: 2026
Type: conference-paper
Abstract: Human emotion perception is inherently context-dependent, especially when observable behaviour is ambiguous. Vision-based emotion recognition systems learn to mimic emotion recognition from data. However, collecting a representative dataset is challenging due to the lack of consensus on what constitutes relevant contexts, as well as ethical and privacy constraints. This brings into question if such systems perform well across diverse real-world contexts. We consider three sources of context based on psychological theory: who expresses the emotion, where the emotion occurs, and how the data is collected. Our key contributions are: (a) Operationalising these contextual variations using EMOTIC, a widely used dataset; (b) Evaluating the supervised EMOTIC model when trained on one subgroup (e.g., women) and tested on another (e.g., men) within each contextual dimension (e.g. gender); and (c) Analysing the zero-shot performance of open-source (vision) language models under the same variations. Our results show that the EMOTIC model exhibits statistically significant performance gaps across contextual subgroups. We find large effect sizes for multiple metrics. These effects remain after controlling for dataset size, and some gaps even increase as the training sets grow. Zero-shot models underperform compared to EMOTIC, and also do not show consistent performance across contexts. These performance gaps are problematic for socially consequential settings, such as education, particularly regarding algorithmic discrimination and bias. Our work stresses the importance of context-aware data collection and evaluation for automated emotion recognition systems.
Authors: Sayak Mukherjee, Tom Viering, Bernd Dudzik
Institution: Delft University of Technology
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7215712818