Year: 2025
Type: preprint
Abstract: Audio-driven portrait animation aims to synthesize realistic and natural talking head videos from an input audio signal and a single reference image. While existing methods achieve high-quality results by leveraging high-dimensional intermediate representations and explicitly modeling motion dynamics, their computational complexity renders them unsuitable for real-time deployment. Real-time inference imposes stringent latency and memory constraints, often necessitating the use of highly compressed latent representations. However, operating in such compact spaces hinders the preservation of fine-grained spatiotemporal details, thereby complicating audio-visual synchronization RAP (Real-time Audio-driven Portrait animation), a unified framework for generating high-quality talking portraits under real-time constraints. Specifically, RAP introduces a hybrid attention mechanism for fine-grained audio control, and a static-dynamic training-inference paradigm that avoids explicit motion supervision. Through these techniques, RAP achieves precise audio-driven control, mitigates long-term temporal drift, and maintains high visual fidelity. Extensive experiments demonstrate that RAP achieves state-of-the-art performance while operating under real-time constraints.
Source: arXiv (Cornell University)
Authors: Du, Fangyu, Li, Taiqing, Qiao, Qian, Yu, Tan, Zhang, Ziwei, Zhen, Dingcheng, Jia, Xu, Yang, Yang, Yin, Shunshun, Liu, Siyuan
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6979285938
Cites:
Cited by:
Related to: