Year: 2025
Type: preprint
Abstract: Retrieval-Augmented Generation (RAG) improves the reliability of large language models by integrating external knowledge, but serving RAG pipelines efficiently is challenging because requests traverse heterogeneous components spanning LLM inference, databases, and CPU-side processing. We present Harmonia, an end-to-end RAG serving framework that addresses these bottlenecks through (i) a flexible pipeline specification interface for composing custom workflows, (ii) heterogeneity-aware deployment that provisions and configures components as a distributed inference system, and (iii) a closed-loop runtime controller that monitors load and execution progress and reduces SLO violations through request prioritization and auto-scaling. Across four RAG applications, Harmonia outperforms commercial alternatives, improving throughput by more than 2.04x while reducing SLO violations by up to 78.4 percent.
Source: arXiv (Cornell University)
Authors: Agarwal, Saurabh, Hu, Bodun, Pabon, Luis, Lee, Myungjin, Srinivasa, Jayanth, Akella, Aditya
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6891971308