ActionDiffusion: An Action-Aware Diffusion Model for Procedure Planning in Instructional Videos
Year: 2025
Type: conference-paper
Abstract: We present ActionDiffusion - a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account. Our approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action in-formation into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mecha-nism in the noise prediction network to learn the corre-lations between different action steps. We report exten-sive experiments on three instructional video benchmark datasets (CrossTask, Coin, and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning. Codes are available at https://www.collaborative-ai.org/publications/shi25_wacv/
Authors: Lei Shi, Paul Bürkner, Andreas Bulling
Institutions: University of Stuttgart, TU Dortmund University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W4409263628