Year: 2025
Type: dissertation
Abstract: The objective of the thesis was development and evaluation of the Proof of Concept (PoC) version of a RAG (Retrieval-Augmented Generation) chatbot implemented for thesis guidance. The thesis writing process often requires specialized knowledge and personal guidance, but the available guidance resources are limited. The need for an automatic, independent guidance system is obvious. A locally operating RAG application was developed that is not dependent on third-party service providers. The application was implemented in Python, using the Langchain library for information retrieval and response generation, the Streamlit user interface, and ChromaDB for vector database management. Artificial intelligence models were integrated via the Ollama framework, using the large language models. Since it was not possible to implement user testing, the functionality of the application was evaluated using the RAGAs tool, which allows for comparable measurement results. The chat functionality was tested with three different language models (Llama 3.2, Phi 3, Qwen 3), and the answers they generated were evaluated using three judge models (Codellama, Mistral, Qwen 2). The developed RAG chatbot proved to be technically functional at the PoC level, able to handle thesis-related questions and provide guidance using locally installed language models. The testing results show that different combinations of language models and judge models produce varying results in terms of the quality of answers. The study shows that independent RAG chatbots are technically feasible and offer a potential solution for automating thesis guidance. Further development requires more extensive user testing and optimization of the application according to real-world usage situations.
Source: Theseus (Ammattikorkeakoulujen)
Author: Veijanen, Jukka
Institution: JAMK University of Applied Sciences
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7115028554