Year: 2025
Type: dissertation
Abstract: Universities face growing volumes of repetitive student inquiries related to courses, admissions, and procedures. While human support remains essential for complex cases, scalable chatbot systems can streamline routine communication and improve student experience. Although Large Language Models have advanced conversational agents, their static knowledge and hallucination risks limit their reliability in domains requiring accurate, up-to-date, and context specific information, such as academic environments. Retrieval Augmented Generation offers a promising solution by grounding responses in relevant documents. This study systematically tests Retrieval Augmented Generation configurations, including sparse (TF-IDF, BM25), dense (BAAI/bge) and hybrid retrieval methods, reranking, and metadata incorporation. We evaluated generation quality using lightweight LLaMA 3.2 models (1B and 3B), tested across baseline, naïve RAG, and zero-shot prompt engineered variants. Performance was measured through retrieval metrics (Precision@k, Recall@k, MRR@k), semantic similarity (BERTScore, SAS, SemScore), and human evaluations. Results show that hybrid retrieval with reranking and metadata yields the highest precision, while prompt optimization significantly enhances generation quality. Lightweight models demonstrated strong generation performance, suggesting that effective domain specific systems can be achieved with smaller computational requirements. The findings provide a practical framework for developing an efficient chatbot in educational institutions with limited computational resources. Deeper investigation into the effectiveness of smaller models warrants further exploration into the relationship between model architecture, retrieval system design, and performance outcomes.
Author: Fazendeiro, Rodrigo Augusto
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7113271995