Work

When is LLM-based reasoning retrieval worth the cost? Lightweight cost-aware query routing for retrieval-augmented generation

Year: 2026
Type: article
Abstract: Retrieval-Augmented Generation (RAG) systems face a cost–effectiveness trade-off: inexpensive retrievers are efficient, whereas reasoning-based retrievers can improve retrieval quality at substantially higher computational cost. We introduce CARE (Cost-aware Adaptive REtrieval), a lightweight cost-aware binary routing method that decides whether each query should be handled by an inexpensive Hybrid retriever or escalated to the ReSP (Retrieve, Summarize, Plan) reasoning-based retriever. The routing decision is made by a lightweight logistic-regression gate using query-level and inexpensive probe-retrieval features. The evaluation covers two BEIR question-answering (QA) datasets, HotpotQA and Natural Questions (NQ), comprising 7,914,797 documents and 10,857 benchmark queries in total, of which 4097 constitute the fully held-out test sets used for evaluation. Using dataset-specific thresholds selected independently on the corresponding validation splits ( τ = 0.30 for HotpotQA and τ = 0.60 for NQ), CARE achieves utilities of 0.8390 and 0.7596, respectively. These dataset-specific operating points yield a macro-average utility of 0.7993 and a macro-average expensive-branch activation rate of 49.31%. On HotpotQA, CARE approximately matches ReSP in retrieval effectiveness, although its Recall@100 is slightly lower (0.8770 versus 0.8795). Its 0.0007 utility advantage is therefore driven by the routing-cost term rather than by superior retrieval effectiveness, with expensive-branch activation reduced from 100% to 88.79%. On NQ, CARE provides a clearer cost-sensitive benefit, achieving 0.7596 utility with only 9.82% activation.
Language: English
Open Access status: hybrid
Estimated APC paid: $3,670
OpenAlex ID: W7154749722
FWCI: 2.197
Cites: 33
Cited by: 1
Related to: 10