A hybrid machine learning framework for synthetic-augmented multimodal emotion recognition via EEG and facial expressions
Year: 2026
Type: article
Abstract: Emotion recognition from electroencephalography (EEG) and facial expressions is limited by small, imbalanced datasets and by differences between individuals. This study presents a hybrid framework combining synthetic data generation and multimodal fusion, evaluated on unseen participants and subjects. For EEG, federated learning with FedProx and participant-specific batch normalisation was combined with SMOTE (FedS-CNNReS). For facial expressions (CK+), synthetic images were generated with a conditional variational autoencoder (CC-Balanced). Because the two datasets come from different individuals, they were coupled through a shared valence–arousal label space and fused at the decision level with the proposed Contrastive Gated Attention Fusion model (CGA-Fusion). Synthetic EEG feature vectors preserved the performance obtained with real data, and training on synthetic data alone was equivalent to real-data training. Synthetic facial images raised the recall of the minority class (sadness) from 0.43 to 0.75. Fusion improved arousal and three-class recognition over the facial classifier alone (from 0.689 to 0.844 and from 0.765 to 0.888); the advantage of CGA-Fusion over a simple product rule came mainly from probability calibration. When both modalities were trained on synthetic data only, the fused model generalised to real data. These results show that synthetic data can support multimodal emotion recognition, especially for under-represented emotions.
Source: Results in Engineering
Institution: Batman University
Language: English
Topic: Emotion and Mood Recognition
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,940
OpenAlex ID: W7225074058
