Semantic Planning with Large Language Models and Vision-Language Models for Collaborative Robots: A Review
Year: 2026
Type: preprint
Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) now allow robots to translate natural-language instructions and visual observations into executable task plans. Despite these advances, the resulting plans remain incompletely grounded in the robot's capabilities and affordances, in scene geometry and physical constraints, and in the collaborator's changing state. The deficiency is most damaging over long horizons, where a local planning or execution error propagates through the remaining actions and compromises the task without announcing itself. Reliable collaboration therefore demands not only higher per-step accuracy but the means to detect planning uncertainty and constraint violations and to respond to them. This survey reviews semantic planning for collaborative robotics at the interface of foundation-model reasoning, multimodal perception, task and motion planning, and human-robot interaction. We formalize collaborative semantic planning as a constrained grounding problem relating instruction, observation, robot capability, task state, and human state, and assess surveyed approaches against five explicit conditions: groundedness, feasibility, safety, legibility, and calibrated abstention. We organize the field into a taxonomy of architectures, indexed by the depth at which semantics enter the planning and control hierarchy, and argue that capability and inspectability have moved in opposite directions across successive generations of architectures. We then state a reliability decomposition separating per-step accuracy from detection quality, which gives, under an explicit independent-step model, the condition under which improving detection dominates improving accuracy. We close with the open challenges and a roadmap for incorporating uncertainty estimation, plan verification, and recovery into next-generation collaborative architectures.
Source: Research Square
Authors: Osama Ali Khan, Abuzar Ghaffari
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204958905