Realistic Speech-Driven Talking Face Synthesis via Audio-Visual Association Exploitation and its Applications
Year: 2024
Type: dissertation
Abstract: Talking face synthesis holds significant importance across various entertainment applications, including digital human animation, visual dubbing in films, and rapid production of short videos.To promote realistic video generation, we propose to fully exploit the implicit information of human speech in talking face synthesis system.In contrast to previous works that borrow external images, videos or labels as reference source to explicitly provide speaker characteristics like facial appearance or talking status, our proposed framework attempt to directly infer these underlying features from human voice, thereby increasing realness and naturalness of the synthesized videos.However, this task is particularly challenging due in two aspects.1) These information within human voice is incomplete or even ambiguous.Such uncertainty brings difficulty to development of talking face system because capturing the required information underlying human speech is non-trivial.2) Successfully driving a talking face requires delicately integration of interleaved information such as speech content or speaker characteristics from human voice.This further increases the complexity of this task.To address the above challenge, we propose to leverage the explicit visual cues as a bridge to promote audio-visual association learning, thus facilitating extensive exploitation of human speech.Specifically, our framework is divided into three components.To ensure that the generation progress is well delegated to each specific semantic space and prevent undesirable interference among them, we first carefully derive a strategy to disentangle the * Doctoral Dissertation,
Authors: Sun, Yasheng, Sun, Yasheng
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7043048241