Year: 2024
Type: article
Abstract: The adaptation of large language models, such as GPT-4O, for educational settings is becoming increasingly relevant, given their potential to improve automated support in learning environments. This study focuses on evaluating five distinct adaptation methods: the standard model (without adaptation), Fine-Tuning, Prompt Engineering, Retrieval-Augmented Generation (RAG), and agent-based systems. These methods were compared to find the most effective for enhancing GPT-4O’s performance in educational tasks. The analysis was conducted using key evaluation metrics such as precision, accuracy, F1-Score, average tokens per response, hallucination rate, and a final comprehensive score. The experimental results indicate that Fine-Tuning, which involves additional training on domain-specific educational data, offers the most significant improvements in terms of accuracy and reduction of erroneous responses (hallucinations). Fine-Tuning is especially effective in educational tasks requiring high accuracy and contextual understanding. Retrieval-Augmented Generation showed promising results by leveraging external data to enhance accuracy and lower hallucinations, making it suitable for tasks needing up-to-date information. Prompt Engineering provided faster response times but had more inaccuracies due to reliance on optimal query formulation without retraining. Agent-based systems excelled in handling complex tasks, though they showed a slight increase in hallucination rates due to their dynamic nature. The baseline performance of the standard GPT-4O model highlighted its limitations, like reduced accuracy and higher hallucination rates, especially in educational contexts. These findings underscore that Fine-Tuning is the most effective adaptation method for educational tasks, offering substantial improvements in accuracy and reliability. Overall, this research highlights the necessity of selecting the appropriate adaptation method based on the specific requirements of educational tasks. The study contributes to the ongoing optimization of large language models for use in educational environments, ensuring that responses are both reliable and contextually relevant.
Source: Municipal economy of cities
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: diamond
OpenAlex ID: W4405770197