Work

Contrastive Sequential-Diffusion Learning: Non-Linear and Multi-Scene Instructional Video Synthesis

Year: 2025

Type: conference-paper

Abstract: Generated video scenes for action-centric sequence descriptions, such as recipe instructions and do-it-yourself projects, often include non-linear patterns, where the next video may need to be visually consistent not with the immediately preceding video but with earlier ones. Current multi-scene video synthesis approaches fail to meet these consistency requirements. To address this, we propose a contrastive sequential video diffusion method that selects the most suitable previously generated scene to guide and condition the denoising process of the next scene. The result is a multi-scene video that is grounded in the scene descriptions and coherent w.r.t. the scenes that require visual consistency. Experiments with action-centered data from the real world demonstrate the practicality and improved consistency of our model compared to previous work. Code and examples available at https://github.com/novasearch/CoSeD

Source: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

DOI: 10.1109/wacv61041.2025.00456

Authors: Vasco Ramos, Yonatan Bitton, Michal Yarom, Idan Szpektor, Joao Magalhaes

Institutions: Laboratório para a Ciência da Computação e Informática, Universidade Nova de Lisboa, Google (United States)

Language: English

Topic: Generative Adversarial Networks and Image Synthesis

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: NOVA Laboratory for Computer Science and Informatics, Fundação para a Ciência e a Tecnologia, Google Research

Awards: G6348741101, G8195625483, NOVA Laboratory for Computer Science and Informatics