Work

Multimodal Emotion Recognition Using Synchronized Facial Expression and Pupillometric Signals

Year: 2025
Type: dissertation
Abstract: This thesis explores advanced methods for recognizing human emotions by analyzing facial expressions and pupillary responses. While facial expressions can be consciously masked, pupillary changes often reflect genuine emotional states. To improve detection accuracy, this research combines both modalities using feature-level (early) and probability-level (late) fusion techniques. Due to the absence of publicly available datasets with synchronized facial and pupillometry data, two datasets were collected: one using emotionally rich video stimuli and another simulating user interactions within abanking application. The study compares general and personalized models to account for individual and contextual differences. Results show that combining facial and pupillary data improves emotion recognition accuracy by 7–13% compared to using either source alone. This work establishes a baseline for future research in adaptive and emotionally aware human-computer interaction.
Language: English
Open Access status: green
OpenAlex ID: W7139329069
Cites: 82
Cited by:
Related to: 10