Work

Speech-Driven Gesture Generation Using Transformer-Based Denoising Diffusion Probabilistic Models

Year: 2024

Type: article

Abstract: While it is crucial for human-like avatars to perform co-speech gestures, existing approaches struggle to generate natural and realistic movements. In the present study, a novel transformer-based denoising diffusion model is proposed to generate co-speech gestures. Moreover, we introduce a practical sampling trick for diffusion models to maintain the continuity between the generated motion segments while improving the within-segment motion likelihood and naturalness. Our model can be used for online generation since it generates gestures for a short segment of speech, e.g., 2 s. We evaluate our model on two large-scale speech-gesture datasets with finger movements using objective measurements and a user study, showing that our model outperforms all other baselines. Our user study is based on the Metahuman platform in the Unreal Engine, a popular tool for creating human-like avatars and motions.

Source: IEEE Transactions on Human-Machine Systems

DOI: 10.1109/thms.2024.3456085

Authors: Bowen Wu, Chaoran Liu, Carlos Toshinori Ishi, Hiroshi Ishiguro

Institutions: The University of Osaka, RIKEN, Advanced Telecommunications Research Institute International

Language: English

Topic: Human Motion and Animation

Subfield: Control and Systems Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $2,495

Funder: Japan Science and Technology Agency