Exploring the performance of large language models in novel command-based tasks
Year: 2025
Type: article
Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language tasks, yet their performance in structured command-based environments remains understudied. This paper evaluates seven leading LLMs—GPT-4o mini, GPT-o1, Deepseek-V3, Deepseek-r1, Claude 3.5 Sonnet, Gemini 2.0 Flash, and Llama 3—on two novel command-based tasks: a pixel painting application requiring precise syntax adherence and a maze navigation task demanding spatial reasoning. Our comprehensive evaluation framework assessed syntax adherence, visual accuracy, and command efficiency across multiple iterations. Results revealed significant performance disparities between models, with notable variations in their ability to interpret commands accurately and reason spatially. Only one model successfully completed all maze navigation tasks, while performance on pixel painting revealed a clear hierarchy in both accuracy and efficiency. This stark performance gap, particularly in spatial reasoning, highlight limitations in most current LLMs’ ability to execute sequential, rule-based commands in structured environments. Our findings contribute to understanding the boundaries of LLM capabilities in command interpretation and procedural reasoning, with implications for applications in robotic control, programming assistance, and interface automation. We propose several directions for future research, including expanded task diversity, architectural analysis, and specialized fine-tuning approaches.
Source: Franklin Open
Authors: Rohan Poudel, Pollob Mollick Raza, Anand Patwa, Gungun Kumari, Akash Shah, Sharik Ahmad
Institutions: Sharda University, Amity University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,900
OpenAlex ID: W4417000540