Beyond Vector Similarity: A Two-Stage Safety-Aware Semantic Caching Architecture for Large Language Model Gateways
Year: 2026
Type: preprint
Abstract: Large Language Model (LLM) gateways rely heavily on semantic caching to reduce inference latency and computational cost by reusing responses for semantically similar queries. However, conventional single-stage semantic caches that depend solely on dense vector cosine similarity are structurally vulnerable to false-positive cache hits. High vector similarity frequently fails to distinguish logical negations, antonym substitutions, subtle parameter shifts, and safety-critical boundary conditions—resulting in contextually incorrect or unsafe cached responses being served to users. This paper presents a two-stage safety-aware semantic caching architecture designed for production LLM gateways. Stage 1 performs fast approximate nearest-neighbor retrieval using dense vector embeddings (pgvector) to identify candidate cache entries above an initial similarity threshold. Stage 2 introduces a deterministic safety and intent verification gate that evaluates structural divergence, negation tokens, and policy boundaries before confirming a cache hit. By decoupling fast candidate retrieval from strict semantic equivalence verification, the proposed architecture significantly reduces false-positive cache hits on adversarial and edge-case prompts while maintaining low end-to-end latency overhead.
Author: A, MappillaiMeeran
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225759770
Cites:
Cited by:
Related to: