DanceCrafter: A two-stage framework for high-quality music-driven image animation
Year: 2026
Type: article
Abstract: Generating dance videos from music and a single reference image poses significant challenges, including realistic motion synthesis, precise rhythmic alignment, and detailed identity preservation. Existing methods often fall short in producing long yet consistent motion sequences over time, while also preserving the visual identity and fine-grained details of the humans. To address these challenges, we propose DanceCrafter, an innovative two-stage framework that integrates two models for pose generation and human animation, respectively. In the first stage, we propose a Vector Quantized Variational Autoencoder (VQ-VAE) model that generates detailed, rhythmically aligned pose sequences directly from input music. In the second stage, we employ a diffusion-based approach to animate a human image with these synthesized poses, resulting in coherent, realistic, and visually appealing human dance videos. Furthermore, to enhance synchronization with the musical rhythm, we introduce a novel beat alignment loss ensuring accurate alignment with musical beats. For the image animation model, we introduce a region enhancement strategy that assigns greater focus to hand and facial areas to improve the details generation. Extensive experimental evaluations demonstrate that our method outperforms existing approaches in rhythmic coherence and human identity consistency.
Authors: Yiwen Wang, Ruiyan Wang, Zhengxue Cheng, Li Song
Institution: Shanghai Jiao Tong University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,450
OpenAlex ID: W7213463527