Work

Speech2Face3D: A Two‐Stage Transfer‐Learning Framework for Speech‐Driven 3D Facial Animation

Year: 2025

Type: article

Abstract: ABSTRACT High‐fidelity, speech‐driven 3D facial animation is crucial for immersive applications and virtual avatars. Nevertheless, advancement is impeded by two principal challenges: (1) a lack of high‐quality 3D data, and (2) inadequate modelling of the multi‐scale characteristics of speech signals. In this paper, we present Speech2Face3D, a novel two‐stage transfer‐learning framework that pretrains on large‐scale pseudo‐3D facial data derived from 2D videos and subsequently finetunes on smaller yet high‐fidelity 3D datasets. This design leverages the richness of easily accessible 2D resources while mitigating reconstruction noise through a simple temporal smoothing step. Our approach further introduces a Multi‐Scale Hierarchical Audio Encoder to capture subtle phoneme transitions, mid‐range prosody, and longer‐range emotional cues. Extensive experiments on public 3D benchmarks demonstrate that our method achieves state‐of‐the‐art performance on lip synchronization, expression fidelity, and temporal coherence metrics. Qualitative user evaluations validate these quantitative improvements. Speech2Face3D is a robust and scalable framework for utilizing extensive 2D data to generate precise and realistic 3D facial animations only based on speech.

Source: IET Image Processing

DOI: 10.1049/ipr2.70155

Authors: Liming Pang, Zhi Zeng, Yahui Li, Guixuan Zhang, Shuwu Zhang

Institutions: Beijing University of Posts and Telecommunications, Chinese Academy of Sciences, Institute of Automation

Language: English

Topic: Human Motion and Animation

Subfield: Control and Systems Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,740

Funder: National Key Research and Development Program of China