Work

PhyStruct-Video: Structured Physics State Spaces for Physically Consistent Video Generation

Year: 2025
Type: preprint
Abstract: Generative video models have recently achieved impressive progress, powered by advances in deep architectures, large-scale datasets, and stable training objectives [1, 2, 3, 4, 5, 6, 7, 8]. Diffusion-based video generators in particular can synthesize long, coherent and high-fidelity videos from text or visual conditions [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. However, most existing methods treat videos as purely statistical sequences and completely ignore the underlying physical laws governing motion and interaction. As a result, generated sequences often exhibit artifacts such as volume-changing “incompressible” fluids, interpenetrating rigid bodies, and non-monotonic mechanical energy in closed systems [21, 22, 23, 24]. These artifacts limit the applicability of generative videos in physics-sensitive domains such as robotics [25, 26], autonomous driving [27], simulation-based design, and scientific visualization [28]. In this paper, we argue that a key obstacle is the lack of an explicit separation between physical state and visual appearance. We propose PhyStruct-Video, a physics-aware video generation framework that introduces a structured physics state space sitting between the conditioning signal and a powerful latent diffusion video decoder [8, 7]. Instead of directly constraining high-dimensional video latents, PhyStruct-Video maintains a scene-level representation consisting of particles or patches, each endowed with 3D position, velocity and material label. A graph-based dynamics prior learns to evolve these states under approximate conservation laws, trained by distillation from differentiable teacher simulators [29, 30, 31, 32, 33, 34, 35] and PDE-inspired residual penalties [21, 22, 36, 24, 37]. A latent diffusion decoder then renders high-resolution videos from the evolving states, with a light motion-consistency term that encourages agreement between simulated velocities and rendered optical flow [38, 39, 40]. We conduct extensive experiments on synthetic physics benchmarks (including fluid, rigid and elastic interactions), real-world physics-centric subsets of WebVid10M, Something-Something V2, and Kinetics-600, as well as text-to-video prompts emphasizing physical interactions. PhyStruct-Video substantially improves physically motivated metrics (e.g., divergence, rigid distance preservation, elastic energy stability) over strong video diffusion baselines, while matching or improving perceptual quality metrics such as FVD, FID and LPIPS [41]. We further show that policies trained on videos generated by PhyStruct-Video exhibit smaller sim-to-real gaps in MuJoCo and CARLA-like tasks [42, 25, 27, 26]. Our results suggest that explicitly modeling structured physics state spaces is a promising direction for making video generation both visually compelling and physically credible.
Author: Junyi Li
Language: English
Open Access status: green
OpenAlex ID: W4417199831
Cites: 36
Cited by:
Related to: 10