MANIPDREAMER: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance
Year: 2026
Type: conference-paper
Abstract: Recent advancements in robotic manipulation video synthesis have shown promise, but challenges remain in achieving robust instruction-following and high visual fidelity. Existing approaches like RoboDreamer rely on linguistic decomposition for compositional understanding but often overlook sequential dependencies and key visual cues such as depth and semantics, which are crucial for improving visual quality. We propose ManipDreamer, an advanced world model based on an action tree structure and enriched with visual guidance. Each instruction is represented as a structured action tree, with nodes linked to learned embeddings that improve compositional reasoning. These embeddings condition the world model, enhancing instruction-following accuracy. To improve visual fidelity, we integrate depth and semantic information via a visual guidance adapter compatible with the world model. ManipDreamer thus enhances both instruction-following accuracy and visual realism. Evaluations on standard benchmarks show ManipDreamer outperforms RoboDreamer, increasing PSNR from 19.55 to 21.05, SSIM from 0.7474 to 0.7982, and reducing Flow Error from 3.506 to 3.201 on unseen tasks. It also boosts the average success rate in robotic manipulation tasks by 2.5% across six RLbench tasks.
Authors: Ying Li, Xiaobao Wei, Xiaowei Chi, Yuming Li, Zhongyu Zhao, Hao Wang, Ningning Ma, Ming Lu, Shanghang Zhang
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4415065902
FWCI:
Cites:
Cited by:
Related to: