Work

Mitigating Hallucination in Large Language Model Code Generation for Higher Education : An Evaluation of Retrieval Augmented Generation

Year: 2024
Type: dissertation
Abstract: Generative AI has gained a rapidly increasing focus in programming higher education, where the broad spectrum of subjects covered by modern Generative AI tools has made them a vital source of accurate and dependable information. However, a critical concern with Generative AI lies in its tendency to produce inaccurate or fictional information. This tendency, often termed “”hallucination, is a significant challenge for Generative AI. Especially in the field of programming higher education since it can lead to ’students inaccurate knowledge acquisition and ultimately affect their learning process negatively. The present study evaluated one of the popular hallucination mitigation techniques, namely Retrieval Augmented Generation on ChatGPT and Gemini. In order to maximise their proficiency in solving programming assignments across varying difficulty levels. An experiment was conducted where ChatGPT and Gemini were tested on 100 randomly selected programming problems provided by Kattis, an automatic software grading tool for computer science programs often used in higher education. ChatGPT and Gemini attempted to solve each assignment, both not using Retrieval Augmented Generation, and using it, the solutions generated were automatically evaluated by Kattis. The results demonstrated an average 10% increase in acceptance rates for both models when using Retrieval Augmented Generation. Furthermore, using Retrieval Augmented Generation led to a lower error rate as well as a higher rate of improvement on previous attempts. The results contribute to the ongoing research on the utility of Generative AI-powered tools in programming education.
Author: Asbai, Ali
Language: English
Open Access status: green
OpenAlex ID: W7011367249
Cites: 55
Cited by: 4
Related to: 10