Design and evaluation of a Retrieval-Augmented Generation system for Maxtech Support Portal : Comparing Vertex AI, Qdrant, and Pinecone architectures in a Finnish-language enterprise context
Year: 2026
Type: dissertation
Abstract: This thesis investigates the creation, implementation and assessment of a Retrieval Augmented Generation (RAG) system specifically designed for the Maxtech Support Portal. The work addresses a real-world enterprise problem whereby users frequently open support tickets that are already documented and addressed on the portal, but users are still struggling to locate relevant information independently. Three different RAG systems were conceptualized and subsequently comparatively tested against each other, one based on the Google Vertex AI integrated solution and two fully custom developed solutions that were tested using two well - established cloud vector databases: Qdrant Cloud and Pinecone. The systems were evaluated by focusing on the component-level evaluation: metadata - filtering, reranking of results, question augmentation and enrichment of documentation with image data at ingestion time, all contributing to answer generation. The RAG Evaluation Dashboard was created to support controlled blind evaluation, structured human scoring, and descriptive analysis. A total of 30 real support questions were picked to go through human assessment, making a total of 270 ratings over three architectures and three evaluators. Responses were rated based on five quality dimensions: accuracy, completeness, faithfulness, relevance, and usefulness. The results show that each of the three architectures was technically feasible to support enterprise automation but with distinctly different quality profiles. Vertex AI provided the most stable overall performance and achieved the highest scores on completeness, faithfulness, and usefulness, while Pinecone achieved the highest accuracy score. The study further demonstrated that metadata filtering and retrieval optimization techniques affected response quality in different practical ways on response quality. A key finding of the research is that evaluating RAG answers is not only a technical task but also a human judgement task. Evaluators sometimes gave different scores to the same answers, especially when the answer was correct but incomplete, or when the system correctly stated that information was not available in the support portal.
Source: Theseus (Ammattikorkeakoulujen)
Author: Madusanka, Jasintha
Institution: Oulu University of Applied Sciences
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162725425