Design and Performance Evaluation of LLM-Based RAG Pipelines for Chatbot Services in International Student Admissions
Year: 2025
Type: article
Abstract: Recent advancements in large language models (LLMs) have significantly enhanced the effectiveness of Retrieval-Augmented Generation (RAG) systems. This study focuses on the development and evaluation of a domain-specific AI chatbot designed to support international student admissions by leveraging LLM-based RAG pipelines. We implement and compare multiple pipeline configurations, combining retrieval methods (e.g., Dense, MMR, Hybrid), chunking strategies (e.g., Semantic, Recursive), and both open-source and commercial LLMs. Dual evaluation datasets of LLM-generated and human-tagged QA sets are used to measure answer relevancy, faithfulness, context precision, and recall, alongside heuristic NLP metrics. Furthermore, latency analysis across different RAG stages is conducted to assess deployment feasibility in real-world educational environments. Results show that well-optimized open-source RAG pipelines can offer comparable performance to GPT-4o while maintaining scalability and cost-efficiency. These findings suggest that the proposed chatbot system can provide a practical and technically sound solution for international student services in resource-constrained academic institutions.
Source: Electronics
Authors: Maksuda Khasanova Zafar kizi, Youngjung Suh(Conceptualization, Formal analysis, Methodology, Project administration, Software, Supervision, Validation, Writing – review & editing)
Institution: Kongju National University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: retrieval-augmented generation, large language models, chatbots, international student admission, dense retrieval, maximum marginal relevance, hybrid retrieval, semantic chunking, open-source LLMs, GPT-4o, answer faithfulness, answer relevance, response latency, resource-constrained institutions
SDG: Quality education
Open Access status: gold
Estimated APC paid: $2,896
Award: G8279346340
OpenAlex ID: W4412907229
