Co-speech holistic 3D motion generation with style from video
Year: 2026
Type: article
Abstract: Speech-driven 3D motion generation has garnered research attention. However, it faces significant challenges in achieving style controllability, due to the scarcity of motion style annotations. To address this, we propose a novel diffusion-based framework for co-speech holistic motion generation that enables example-based style control from videos. Our approach integrates hierarchical speech encoding with rhythm-aware denoising to produce natural and synchronized gestures and expressions. For effective style guidance, we introduce a contrastive style encoder that captures discriminative style representations from reference clips without explicit labeling, enabling generalization to motion styles unseen during training. Furthermore, we design a neural mapper that aligns 2D and 3D gesture features in a shared embedding space, facilitating direct style extraction from in-the-wild videos and seamless transfer to 3D motion. Extensive experiments and user studies show that our proposed approach achieves state-of-the-art performance in both qualitative and quantitative evaluations, offering a flexible solution for controllable motion generation.
Source: Graphical Models
Institution: Beijing Jiaotong University
Language: English
Topic: Human Motion and Animation
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,180
OpenAlex ID: W7162687643