Work

Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations

Year: 2026
Type: article
Abstract: Retrieval-augmented generation (RAG) is commonly assessed on benchmarks, but its behavior under employees’ workplace questions remains less clear. We compared single-pass, agentic, and hybrid RAG in a Slovenian public-sector assistant to examine user-perceived quality, actual tool use, and token consumption. Configurations were randomized per turn, and their names were not shown to users. We analyzed 701 responses from 131 accounts; 402 answers were evaluated by 73 accounts using five 1–5 rating items and a separate binary usefulness judgment. Resource use was application-recorded input plus output tokens, excluding re-ranking. Hybrid had the highest descriptive mean quality (4.15), followed by Single-pass (3.93) and Agentic (3.79), but neither dimension tests nor adjusted composite contrasts established superiority. Hybrid also had the highest mean token total among rated responses (41,143 versus 32,977 and 28,432). Agentic initiated document retrieval on 62.9% of turns; Hybrid used additional retrieval in 3.5%. Thus, the comparison showed a design–behavior gap without establishing an escalation benefit. The study provides field-evaluation evidence and practical deployment recommendations; incomplete rating coverage, configuration-specific prompts, and mixed conversation histories limit generalization and causal interpretation.
Language: English
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7236733916
FWCI:
Cites:
Cited by:
Related to: 10