Work

DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion

Year: 2026

Type: article

Abstract: Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, fixed multimodal conditioning can let dense rhythmic and acoustic cues dominate sparse semantic cues, weakening rare content-specific gestures. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. DynaConTalk performs diffusion in stationary wavelet transform (SWT) coefficient space, where temporally aligned bands separate coarse posture evolution, gesture strokes, and high-frequency expressive details. We further introduce dynamic proposal-consensus conditioning to adaptively balance rhythmic, acoustic, semantic, textual, and identity/activity cues according to the current speech context and denoising state. Finally, to ensure smooth transitions between windows and support flexible user control, we re-noise any reference signal - whether a keypose, a motion clip, or a short boundary condition - to match the current diffusion timestep before injecting it into the denoiser. This unified strategy allows seamless long-form generation, localized editing, and style-driven guidance to be handled within a single coherent process, without task-specific modules or post-hoc stitching. Experiments show that DynaConTalk improves gesture realism, facial accuracy, temporal continuity, and controllable editing over prior holistic co-speech generation methods. Code and data for this paper are at https://zhuyifeiabcd1.github.io/DynaConTalk/.

Source: ACM Transactions on Graphics

DOI: 10.1145/3842553