Year: 2025
Type: conference-paper
Abstract: Diffusion based video generation has received significant attention in both the academic and industrial communities. Despite recent exploration of diverse conditional inputs for better video generation control, existing methods, primarily targeting individual tasks, often fall short in real-world scenarios where users may use any form of conditioning, either individually or combined. To address this, we propose a Unified-modal Video Generation system capable of handling multiple video generation tasks across different modalities. Our approach introduces the concept of generative freedom in the diffusion process, which allows us to reclassify video generation tasks into high-freedom and low-freedom categories based on the solution space given certain conditions. We then design different diffusion paradigms for each category. For high-freedom video generation, we present a base model that is capable of handling varied semantic combinations of text and image. For low-freedom video generation, we propose the Biased Gaussian Noise (BGN) to tackle the discrepancy of the diffusion process between the training and inference stage when using strong conditional guidance strategy. Our proposed UniVG achieves superior objective results on public datasets, surpassing the current open-source methods and is on par with the current close-source method Gen2 and Pika in human evaluations. For more samples, visit our homepage.
Authors: Ludan Ruan, Lei Tian, Chuanwei Huang, Xu Zhang, Xinyan Xiao
Institution: Baidu (China)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4415708362