Scalable Audiovisual Masked Autoencoders for Efficient Affective Video Facial Analysis
Year: 2025
Type: article
Abstract: Affective video facial analysis (AVFA) has emerged as a key research field for building emotion-aware intelligent systems, yet this field continues to suffer from limited data availability. In recent years, the self-supervised learning (SSL) technique of masked autoencoders (MAEs) has gained momentum, with growing adaptations in audiovisual contexts. While scaling up has proven essential for breakthroughs in general multimodal learning domains, its specific impact on AVFA remains largely unexplored. Another core challenge in this field is capturing both intra- and intermodal correlations through scalable audiovisual representations. To tackle these issues, we propose AVF-MAE++, a family of audiovisual MAE models designed to efficiently investigate the scaling properties in AVFA while enhancing cross-modal correlation modeling. Our framework introduces a dual masking strategy across audio and visual modalities and strengthens modality encoders with a more holistic design to better support scalable pretraining. Additionally, we present an iterative audiovisual correlation learning module that improves correlation learning within the SSL paradigm, addressing the limitations of previous methods. To support smooth adaptation and reduce overfitting risks, we introduce a progressive semantic injection strategy, organizing the model training into 3 structured stages. Extensive experiments conducted on 17 datasets, covering 3 major AVFA tasks, demonstrate that AVF-MAE++ achieves consistent state-of-the-art performance across multiple benchmarks. Comprehensive ablation studies highlight the importance of each proposed component and provide deeper insights into the design choices driving these improvements. Our code and models have been publicly released at Zenodo.
Source: Intelligent Computing
Authors: Xuecheng Wu, Junxiao Xue, Xinyi Yin, Yunyun Shi, Liangyu Fu, Danlei Huang, Yifan Wang, Jia Zhang, Jiayu Nie, Jun Wang
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W4416720874