Work

Grounding Language Models at Scale

Year: 2026
Type: dissertation
Abstract: Large language models and vision-language models now achieve impressive performance on a wide range of digital tasks, yet they still struggle in physically situated, non-digital settings where language must be grounded in perception, interaction, and action. At the same time, the notion of grounding is used inconsistently across AI and NLP, and we lack a clear understanding of when and how grounding emerges in current models, or how to scale it in ways that are both engineering-feasible and informative about human language learning. This thesis investigates language grounding in language models at scale, guided by two prerequisites: methods must be compatible with large-scale training and open-world data, and models should incorporate human-inspired constraints in a controlled, mechanistic way. First, I show that grounding can emerge: in a minimal, CHILDES-inspired testbed that separates environmental from linguistic tokens, models trained with a standard next-token objective behaviorally ground symbols, which I trace to specific middle-layer attention heads, with cross-architecture comparisons showing where such grounding does and does not arise. Second, for grounding to perception, I introduce Grounded Open Vocabulary Acquisition and an object-centric vision-language model, OctoBERT, that jointly performs language modeling and word-to-object grounding, enabling data-efficient learning of both seen and unseen grounded words. Third, for grounding to feedback, I propose Trial-and-Demonstration, a framework combining student trials, teacher demonstrations, and trajectory-aware rewards, and show that corrective feedback reshapes word-specific learning dynamics and improves efficiency. Finally, for grounding to action, I train a joint world-language model over interleaved images and action descriptions, so that predicting how an action transforms a scene becomes a learning signal for word meaning, with the largest gains for result verbs. Together, these contributions provide a mechanistic account of emergent grounding and introduce scalable, human-inspired training regimes that ground language models to perception, feedback, and action, advancing both practical grounded AI and computational theories of language acquisition.
Author: Ma, Martin
Language: English
Open Access status: green
OpenAlex ID: W7221068889
Cites: 213
Cited by:
Related to: 10