Robot Learning Through World Models
Year: 2026
Type: dissertation
Abstract: Embodied AI introduces a new generation of intelligent agents capable of closed-loop perception and control in real-world environments. However, despite recent progress in robot learning, Most current approaches to robot manipulation are heavily based on supervised learning frameworks. Methods like imitation learning require a large number of examples to succeed in prediction and control. These examples are usually collected by teleoperation, which is expensive to collect, difficult to scale, and prone to bias, limiting the scalability and generalizability of these systems across tasks, environments, and embodiments. On the other hand, humans accumulate knowledge and experience by building internal models of how the world around them works. These world models allow them to predict the future outcomes of their potential actions, reducing the trial and error in the real world needed to learn successful behaviors. This thesis focuses on the problem of equipping robots with the ability to understand the world, learn action representations, imagine the future, and make decisions based on their predictions. We introduce methods for both pretraining and post-training of imitation learning models using world models, where we improve these models without any physical robot interactions. Pretraining in a self-supervised way, by learning latent action representations from large-scale unlabeled human video data, sourced from the Internet, performing actions with everyday objects. Then, training on downstream tasks of a target embodiment using a paired image action dataset using supervised learning. Finally, post-training in the sense of test-time decision-making, given the trained imitation learning model, a learned world model predicts different future states, and a reward model evaluates these imagined outcomes to select the highest-value sequence for action execution. Our experiments show that the proposed methods outperform previous approaches without world models on LIBERO and CALVIN benchmarks and also on real-world experiments, not only by requiring fewer examples to learn, but also by achieving higher performance. The proposed methods open up the potential for building foundation models for robotics by pretraining large models on much larger web-scale video data, and that are able to think and reason in their own imagination to adapt to environments.
DOI: 10.82419/356
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7164150232