Work

The Framework Ranked First on All 28 Questions and Still Lost Two of Three Hypotheses: A Pre-registered Comparison of LlamaIndex 0.14.25 Defaults with a Production Hybrid Retriever

Year: 2026
Type: preprint
Abstract: Every team that adopts a retrieval framework bets its search quality on defaults it has not measured: chunk size, embedding model, similarity behaviour on questions the corpus cannot answer. This study makes that bet a number. On a fixed corpus of 20 synthetic documents and a frozen set of 40 questions (28 in-corpus, 12 adversarial), LlamaIndex 0.14.25 under all framework defaults (VectorStoreIndex, SimpleVectorStore, local all-MiniLM-L6-v2 embeddings) was compared against the retrieval pipeline already built and measured in the author's rag-foundation production project (a BM25 plus TF-IDF hybrid with reciprocal rank fusion and a coverage rerank), under hypotheses and verdict lines frozen in a pre-registration committed before any measured run. A harness integrity gate ran first: the production arm reproduced the sibling study's measurement exactly (top-1 hit on 23 of 28), so the comparison measures the framework, not the harness. LlamaIndex ranked the ground-truth document first on all 28 in-corpus questions (top-1 1.0000 against 0.8214; top-5 1.0000 against 0.9643; MRR 1.0000 against 0.8929), yet H1 was REFUTED, because the frozen line was the top-5 margin and the difference of 0.0357 sits inside the 0.10 tie band: the production hybrid was already at 27 of 28. On adversarial questions H2 was CONFIRMED: under a pre-registered own-floor rule (each arm held to the 10th percentile of its own genuine-match scores), the production arm spuriously retrieved on 9 of 12 questions (0.7500) and LlamaIndex on 5 of 12 (0.4167), a difference of 0.3333 against a 0.25 line. The difference is in the scores, not the picks: the two arms name the same top-1 document on 7 of the 10 adversarial questions where both return anything. H3 was REFUTED by a wide margin: the LlamaIndex build took a median of 1.5073 s against 0.0028 s for the production build, a per-document ratio of 537.8 against a registered band of 0.5 to 2.0, and median query latency was 0.0738 s against 0.0005 s. The claims are deliberately narrow: one small synthetic corpus, one embedding model, one machine, one scored pass. Within those bounds, the framework's defaults bought a perfect top-1 ranking and materially fewer confident wrong retrievals, at roughly five hundred times the indexing cost, and no vendor page states any of those three figures for a buyer's own corpus.
Language: English
Open Access status: green
OpenAlex ID: W7236159774
Cites:
Cited by:
Related to: 10