Work

Impact of Developer Queries on the Effectiveness of Conversational Large Language Models in Programming

Year: 2025

Type: article

Abstract: This study investigates the effects of LLM-based coding assistance on web application development by students using a frontend framework. Rather than comparing different models, it focuses on how students interact with LLM tools to isolate the impact of query type on coding success. To this end, participants were instructed to rely exclusively on LLMs for writing code, based on a given set of specifications, and their queries were categorized into seven types: Error Fixing (EF), Feature Implementation (FI), Code Optimization (CO), Code Understanding (CU), Best Practices (BP), Documentation (DOC), and Concept Clarification (CC). The results reveal that students who queried LLMs for error fixing (EF) were statistically more likely to have runnable code, regardless of prior knowledge. Additionally, students seeking code understanding (CU) and error fixing performed better, even when normalizing for previous coding ability. These findings suggest that the nature of the queries made to LLMs influences the success of programming tasks and provides insights into how AI tools can assist learning in software development.

Source: Applied Sciences

DOI: 10.3390/app15126836

Authors: Viktor Taneski, Sašo Karakatič, Patrik Rek, Gregor Jošt

Institution: University of Maribor

Language: English

Topic: Teaching and Learning Programming

Subfield: Computer Science Applications

Field: Computer Science

Domain: Physical Sciences

SDG: Quality education

Open Access status: gold

Estimated APC paid: $2,896