Pengenalan Emosi Manusia Melalui Suara Menggunakan Metode Convolutional Neural Network dan Transformer
Year: 2026
Type: article
Abstract: Communication between humans and machines currently lacks a natural emotional component, since systems generally understand verbal content without capturing the emotional meaning conveyed by tone and timbre. This study aims to design a Speech Emotion Recognition model by integrating Convolutional Neural Network (CNN) and Transformer architectures to capture spatial features as well as long-range temporal dependencies in speech signals. The dataset combines four public corpora, namely RAVDESS, CREMA-D, SAVEE, and TESS, comprising 12,798 audio files across seven emotion classes (angry, disgusted, fearful, happy, neutral, sad, and surprised). The audio signals were processed through Mel Spectrogram feature extraction along with delta and delta-delta coefficients, balanced through data augmentation, and trained using a 5-Fold Stratified Cross Validation scheme. Testing on an independent test set shows that the hybrid model achieves a 5-fold average accuracy of 75.80%, while the best-performing single-fold model reaches 77.00% accuracy with a Macro Average F1-Score of 78.00%, and all emotion classes obtain an AUC value above 0.90. Applying Ensemble Learning (Soft Voting) across the five fold models further improves the testing accuracy to 81.00% with a Macro Average F1-Score of 0.82, representing the highest overall result achieved in this study. These results demonstrate that the integration of CNN and Transformer is effective and generalizes well on heterogeneous speech data.
Institution: Universitas Pendidikan Ganesha
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W7223533313