Year: 2026
Type: article
Abstract: Language produced by a system that has never acted in the world describes that world from the outside. Embodied systems that perceive through their own sensors and change what they perceive by acting offer a different setting for language generation, one in which the meaning of a term can be tied to sensory evidence and to the actions the term licenses. This article examines what such grounding requires and how far current systems supply it. We distinguish three requirements, namely perceptual anchoring of reference, action conditioned meaning and situated deixis, and show that pipelines which caption an image and pass the caption to a planner satisfy only the first. End-to-end architectures that map sensory observations and instructions directly to actions and to language remove the intermediate text bottleneck, and we compare eight such systems by input, output and training signal. We then examine the data available, which is smaller than the corpora used for text pre-training by five orders of magnitude, and the evaluation problem, where reference based caption metrics reward fluency and cannot detect a description of an object that is not present. A taxonomy of grounding failures is proposed together with the diagnostic each one requires, and seven open problems are stated.
Author: Manasy Jayasurya
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Keywords: embodied AI, captioning, training data size
Open Access status: hybrid
OpenAlex ID: W7236487955
