Work

Replanning human–robot collaborative tasks with vision–language models via semantic and physical dual–correction

Year: 2026

Type: article

Abstract: Human–robot collaborative assembly requires robots to interpret ambiguous corrective instructions while producing physically executable motions. Vision–language models (VLMs) provide semantic reasoning but may select logically inconsistent targets or misjudge execution outcomes. We propose a replanning framework that maps human instructions to Action Target candidates, including grasp position and tool selections, and combines an Internal Correction Model for pre-execution logical verification with an External Correction Model for post-execution visual verification. The framework integrates VLM reasoning with 6-DoF grasp generation and collision-free trajectory planning. Simulation ablations show configuration-dependent effects: internal correction improves candidate validity, whereas external correction enables recovery for a low-latency VLM but can reduce success when visual verification produces false negatives. Real-world experiments with an upper-body humanoid robot achieved a 66.7% success rate for grasp position adjustment in the object fixation task; in the tool preparation task, success rates were 100% for initial selection and 75.0% for corrective selection. These results demonstrate interactive replanning across physical and semantic collaborative tasks while identifying visual-state verification as a key limitation.

Source: Advanced Robotics

DOI: 10.1080/01691864.2026.2732745

Authors: Taichi Kato, Takuya Kiyokawa, Namiko Saito, Kensuke Harada

Institutions: The University of Osaka, Microsoft Research Asia (China), National Institute of Advanced Industrial Science and Technology

Language: English

Topic: Robot Manipulation and Learning

Subfield: Control and Systems Engineering

Field: Engineering

Domain: Physical Sciences

Open Access status: hybrid