Year: 2026
Type: dissertation
Abstract: Recent advancements in reinforcement learning (RL) have produced agents with impressive performance in domains such as video and board games. Yet, current RL methods remain datainefficient and generalize poorly to new environments, as they typically learn from scratch without leveraging prior knowledge. This thesis explores how language can improve the sample efficiency and generalization of RL agents by incorporating prior knowledge into their internal world models. It further investigates the use of large language models (LLMs) for black-box optimization (BBO), where an unknown objective must be optimized through trial-and-error, with applications such as prompt optimization and symbolic regression. Finally, these two lines of work converge in programmatic world models, where program synthesis, viewed as a form of BBO, is applied to world modeling. The resulting agents can rapidly adapt to new tasks described in language, while relying on interpretable and verifiable internal models. To begin, the thesis demonstrates how to learn a stochastic world model of an environment exhibiting novel dynamics, accompanied by language descriptions at test time, improving over the state of the art in terms of performance and sample efficiency. Next, LLM-based BBO methods are developed for two tasks: prompt optimization for code, via docstring reformulation, and symbolic regression. The former exposes an unexpected robustness of open-source LLMs to docstring changes, revealing limits of prompt optimization, whereas the latter achieves comparable or superior results to the state of the art while producing simpler formulas. Finally, two approaches for programmatic world models are considered. The first presents a framework to generate code world models via LLMs, together with a method for code generation guided by Monte Carlo Tree Search. The approach achieves superior coding performance over baselines and faster, interpretable planning agents. The second proposes a benchmark for visual planning, to compare approaches that ground symbolic world models to observations with vision-language models (VLMs) and approaches that plan directly with VLMs. The results reveal complementary strengths of symbolic and direct VLM-based planning. Taken together, the thesis contributions mark progress toward autonomous agents that can reason, plan, and optimize through language, combining the sample efficiency and interpretability of modelbased methods with the flexibility of LLMs.
Source: Aaltodoc (Aalto University)
Author: Dainese, Nicola
Institution: Aalto University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155656177