Year: 2026
Type: conference-paper
Abstract: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks.However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance.To bridge this gap, we introduce TURTLEAI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain.Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns.We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving success rates below 30%.To address these limitations, we propose a data generation technique that requires only a small set of seed samples.Fine-tuning Qwen2-VL-72B on the resulting synthetic data yields an improvement of about 20% on realworld tasks.Our failure analysis reveals that GPT-4o struggles with spatial reasoning and precise visual replication, whereas fine-tuning primarily improves the alignment between visual reasoning and code implementation.Pixtral-12B Pixtral-12B-TURTLE (Fine-tuned) GPT-4o Qwen2-VL-72B Qwen2-VL-72B-TURTLE (Fine-tuned)Analyze the provided image of a geometric pattern.Write Python code to replicate the pattern in the image. Input ImageN.A.(a) VLMs' outputs for replicating different input images.def draw(t): """ Draws a flower-like pattern composed of 12 dodecagons.Each dodecagon is rotated by 30 degrees relative to the previous one, creating a star-like structure.""" def draw_polygon(side_length=50, num_sides=12): """ Draws a single regular polygon with the given side length and number of sides.""" angle = 360 / num_sides for _ in range(num_sides): t.forward(side_length) t.left(angle) def draw_flower(num_petals, side_length, num_sides, angle): """ Draws a flower with the given number of petals, side length, number of sides, and angle.""" for _ in range(num_petals): draw_polygon(side_length, num_sides) t.left(angle)
Authors: Chao Wen, Jacqueline Staub, Adish Singla
Institution: Universität Trier
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
Funder: European Commission
OpenAlex ID: W7166808082