Get started
Work

Evaluating the Diversity of LLM-Generated Beginner C Programming Tasks Using Local Ollama Models

Year: 2026
Type: article
Abstract: The increasing capabilities of large language models (LLMs) have enabled new approaches to automated programming task generation for scalable computer science education. While prior research has primarily focused on the correctness and usability of generated tasks, less attention has been given to their diversity and novelty. This study addresses this gap by evaluating the diversity of beginner-level C programming tasks generated by locally deployed LLMs using the Ollama framework. Six 8B-parameter models were evaluated across four prompt categories representing common introductory programming task types, resulting in a dataset of 240 generated tasks. Diversity was assessed using two complementary metrics: task diversity, which captures variation in textual descriptions, and task type diversity, which reflects variation in the underlying problem structure. In addition, prompt sensitivity was analyzed. The results show that all models reliably produce structurally valid outputs, achieving a 100% JSON validity rate. However, substantial differences were observed in diversity across models. While some models generate a higher proportion of unique task descriptions, all exhibit a consistent gap between textual and functional diversity, indicating a tendency toward paraphrasing rather than generating fundamentally new problem types. Prompt formulation also significantly influences diversity. These findings highlight that ensuring meaningful functional diversity remains a key challenge for the effective use of LLMs in programming education.
Institution: J. Selye University
Language: English
Open Access status: diamond
OpenAlex ID: W7217720331
FWCI:
Cites: 5
Cited by:
Related to: 10