Year: 2026
Type: dissertation
Abstract: A limitation of Large Language Models (LLMs) is their tendency to hallucinate, where their output is contradictory with either established world knowledge or context provided in the prompt. The hallucination rate has been previously shown to reduce with the use of In-Context Learning (ICL). ICL is an ability of LLMs to learn new tasks from a few in-context demonstrations. This thesis focuses on detecting faithfulness hallucinations in AI-generated Computer Science education material. In this thesis, a faithfulness hallucination detection pipeline is developed and evaluated with a number of LLMs and various ICL configurations using an existing dataset consisting of AI-generated programming exercises, which is further altered to incorporate a larger number of faithfulness hallucinations. The evaluation compares the ICL configurations with baselines obtained by using the model in a regular, 0-shot prompting setting. The baseline performance of the models are reasonably good, where larger models perform better than smaller models. The results show that with the use of ICL, the evaluated models sometimes improve their performance over the baselines. Furthermore, it is seen that smaller models benefit from ICL disproportionally, experiencing larger performance gains than larger models. The developed pipeline is not able to completely eliminate the performance gap between larger and smaller models, but it is nearly halved. Furthermore, of the small models, Llama-8B obtains performance scores almost as good as the top performer with the use of ICL. The results suggest that with careful configuration, In-Context Learning can be used to improve the performance of Large Language Models in detecting faithfulness hallucinations in AI-generated programming exercises.
Source: Aaltodoc (Aalto University)
Author: Kääriäinen, Aleksi
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204020829