MultiFed-Emote: Privacy-Preserving Multimodal Emotion Recognition via Federated Fusion of Physiological, Speech, and Facial Signals
Year: 2025
Type: conference-paper
Abstract: As human-computer interaction and affective computing advance, accurate emotion detection across diverse modalities remains critical for applications in healthcare, education, and immersive technologies. This work introduces MultiFed-Emote, a privacy-preserving system for multimodal emotion recognition that integrates speech (CREMA-D), facial expressions (FER-2013), and physiological signals (PhyMER) within a federated learning framework. To address challenges of data privacy and modality-specific limitations, we preprocess each dataset using audio standardization, image normalization with augmentation, and EEG/EDA filtering, and extract robust features including MFCCs, CNN embeddings, and Power Spectral Density values. The fused multimodal model uses an attention-driven hybrid approach to dynamically weight modality contributions. Experiments yield accuracies of approximately 67 percent for speech, 62 percent for facial, and 65 to 80 percent for physiological data in centralized settings, with federated fusion targeting 73 to 78 percent. Benchmarking against state-of-the-art models demonstrates competitive performance while preserving privacy, supporting scalable applications in personalized assistants and mental health monitoring.
Authors: Faldu, Kush, Kanabar, Ritu
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6949709458
FWCI:
Cites:
Cited by:
Related to: