Multimodal Emotion Recognition Using Synchronized Facial Expression and Pupillometric Signals
Year: 2025
Type: dissertation
Abstract: This thesis explores advanced methods for recognizing human emotions by analyzing facial expressions and pupillary responses. While facial expressions can be consciously masked, pupillary changes often reflect genuine emotional states. To improve detection accuracy, this research combines both modalities using feature-level (early) and probability-level (late) fusion techniques. Due to the absence of publicly available datasets with synchronized facial and pupillometry data, two datasets were collected: one using emotionally rich video stimuli and another simulating user interactions within abanking application. The study compares general and personalized models to account for individual and contextual differences. Results show that combining facial and pupillary data improves emotion recognition accuracy by 7–13% compared to using either source alone. This work establishes a baseline for future research in adaptive and emotionally aware human-computer interaction.
Source: TSpace (University of Toronto)
Author: Atallah, Jackleen Nagy Haleem
Institution: University of Toronto
Language: English
Topic: Emotion and Mood Recognition
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7139329069
