Learning semantic representations of language, vision and code for visualization-oriented dialogue
Year: 2026
Type: dissertation
Abstract: Data visualization is essential for navigating the vast volume of data that defines our time, letting humans use perception to make sense of complex datasets. Yet creating visualizations requires mastery of specialized programming languages, producing a gap between the analytical intention in the user's mind and its implementation in executable code. This raises the cognitive load on experts, who cannot concentrate fully on visual exploration while also writing the visualization program, and it creates a barrier for non-experts, hindering widespread access to insights from data. Bridging this gap is a central challenge for Visualization-oriented Natural Language Interfaces (V-NLIs), which translate human intent into executable code. This work decomposes the problem into two interrelated challenges: understanding, which builds rich, contextual representations of user intent, and generation, which reliably synthesizes visualization programs under strict real-time constraints. The thesis addresses both through five interconnected contributions. It begins with a survey of 119 papers, revealing that while the community focuses on sophisticated interactive exploration, the underlying technology relies on classical, rule-based methods, motivating a turn to deep learning. KeywordScape, a visual analytics system, makes the contextualized representations of transformer models visible and interactive, showing they can themselves resolve ambiguity and enable fine-grained exploration. Paparazzi extends this to multimodal representations, revealing that large vision-language models such as CLIP exhibit surprising deficiencies in 3D viewpoint understanding, and introduces a data-efficient contrastive fine-tuning method to improve geometric grounding. Finally, VIST5, a practical V-NLI built on a retrieval-augmented architecture around a small language model, shows that complex visualization code can be synthesized in real time while ensuring data privacy and modular extensibility.
DOI: 10.22032/dbt.71264
Author: Voigt, Henrik
Language: English
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7170734941