Optimization of Customer Feedback Summarization Using Large Language Models (LLM) and Advanced Retrieval-Augmented Generation
Year: 2025
Type: article
Abstract: Customer feedback, often shared through online reviews, plays a crucial role in shaping business strategies. However, the overwhelming volume of such reviews poses two major challenges: valuable insights often go unnoticed, and manual analysis introduces human bias. To address this, we propose a system that leverages large language models (LLMs) integrated with the LangChain framework to answer natural language queries over customer reviews. A synthetic dataset was created to resemble food delivery reviews typically seen on the Play Store and was stored in a vector database. On receiving a user query, relevant reviews are retrieved using advanced Retrieval-Augmented Generation (RAG) techniques, namely Hierarchical Chunk Retrieval and RAG Fusion that are further refined using the Declarative Self-improving Python (DSPy) framework to generate accurate, grounded responses. The system was evaluated using LLaMA-3-8B-InstructLite, GPT-3.5-Turbo, and Gemini-1.5-Pro, and compared against two non-LLM baselines: BM25 and a fine-tuned BERT model. Results show that our LLM-based pipeline outperforms by 15% in semantic and factual accuracy. Component-level analysis showed that enhanced retrieval strategies improved semantic relevance by up to 4.9%, lexical coverage by 12.1%, and factual consistency by 9.9% over traditional RAG. Further integration of DSPy led to an additional 10.8% boost in linguistic fluency and a 9.0% gain in factual alignment. Among the evaluated models, Gemini-1.5-Pro combined with RAG Fusion and DSPy produced the most fluent and factually accurate responses, demonstrating the effectiveness of combining hybrid retrieval with LLM-driven reasoning for query-based feedback response system.
Source: IEEE Access
Institutions: University of Florida, R.V. College of Engineering
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W4412352775