Year: 2026
Type: conference-paper
Abstract: We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans.Unlike prior work which focuses mainly on text-only guidance, or treats vision and language in isolation, VIGiA supports grounded, plan-aware dialogue that requires reasoning over visual inputs, instructional plans, and interleaved user interactions.To this end, VIGiA incorporates two key capabilities: (1) multimodal plan reasoning, enabling the model to align uni-and multimodal queries with the current task plan and respond accurately; and(2) plan-based retrieval, allowing it to retrieve relevant plan steps in either textual or visual representations.Experiments were done on a novel dataset with rich Instructional Video Dialogues aligned with Cooking and DIY plans.Our evaluation shows that VIGiA outperforms existing state-of-the-art models on all tasks in a conversational plan guidance setting, reaching over 90% accuracy on plan-aware VQA. 1 1 Model, code and dataset can be found at https:// github.com/dmgcsilva/vigia. Instructional Video PlanUSER: Done, should it be smaller pieces?VIGIA: That looks great!The recipe calls for thin slices, so that looks perfect to sauté.User image VIGIA: Next is Step 7, brush the mustard mixture evenly over the top and sides of the tenderloin.USER: Can you show me how to do that?VIGIA: Sure! Playing the video moment.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140096279