Work

An Adaptive ANN Indexing and Parameter-Tuning Framework for SLA-Constrained Retrieval-Augmented Generation (RAG) Microservices in Relational Vector Databases

Year: 2026
Type: article
Abstract: Enterprise Retrieval-Augmented Generation (RAG) architectures increasingly consolidate operational relational metadata and high-dimensional vector embeddings within unified relational database management systems (RDBMSs) to eliminate cross-system synchronization latency and preserve transactional ACID guarantees. However, executing Approximate Nearest Neighbor (ANN) searches—specifically over Hierarchical Navigable Small World (HNSW) proximity graphs and Inverted File with Flat Compression (IVFFlat) partitions—inside stateless, multi-tenant RAG microservices exposes a severe multi-objective optimization bottleneck. Production retrieval pipelines must simultaneously satisfy strict 99th-percentile (p99) tail-latency Service Level Agreements (SLAs), operate within bounded relational shared-buffer memory pools, and guarantee high top-k semantic recall (Recall@k) under non-stationary query concurrency. Static, heuristically selected index topologies and runtime parameters frequently induce either catastrophic buffer-pool page thrashing when graph structures exceed resident memory or severe queueing-induced SLA violations during burst traffic. To address this fundamental systems challenge, this paper proposes an Adaptive ANN Indexing and Parameter-Tuning Framework (AAIPT) underpinned by a formal buffer-pool-aware analytical cost model and an M/G/c queueing-theoretic predictive admission controller. First, we derive a closed-form page-fault and service-time cost model that couples relational 8-KB buffer page layouts, HNSW random-walk pointer locality, and IVFFlat sequential scan characteristics to select optimal offline index topologies without inducing out-of-core thrashing. Second, we formulate online runtime hyperparameter modulation—governing graph candidate exploration depth (ef_search) and Voronoi cell probing (n_probe)—as a Lyapunov-damped constrained tail-latency minimization problem driven by real-time connection-pool utilization and service-time variance. Third, we conduct an extensive empirical evaluation across three large-scale benchmark workloads (d in {384, 768, 1536}, N up to 2 x 10^6 vectors, and arrival rates up to 300 QPS) comparing AAIPT against five static and heuristic baselines. Experimental results demonstrate that AAIPT reduces p99 tail latency by 38.4% to 52.1%, eliminates out-of-core buffer thrashing spikes of up to 310%, and reduces microservice SLA violation rates from 41.5% down to 0.8% while strictly preserving Recall@10 >= 0.95.
Language: English
Open Access status: green
OpenAlex ID: W7226401138
FWCI:
Cites:
Cited by:
Related to: 10