Query-Variation Robustness of Sparse, Latent-Semantic and Fused First-Stage Retrieval: A Reproducible Pilot Study on the Vaswani Collection
Year: 2026
Type: preprint
Abstract: Query-variation robustness asks whether a retrieval system keeps working when the same information need is worded differently. This paper reports an empirical evaluation across three benchmarks: the classic Vaswani/NPL collection (11,429 abstracts, 93 queries) and modern biomedical collections from BEIR (SciFact and NFCorpus), accompanied by a human audit of 200 stratified query rewrites (Cohen's kappa = 1.000, 100% validity on the analysis set). We compare three first-stage retrieval systems: BM25, a non-neural latent-semantic-analysis (LSA) retriever, and their reciprocal-rank-fusion (RRF) hybrid. Using the four meaning-preserving categories of Penha et al., 644 query rewrites were kept for primary analysis on Vaswani. Misspellings degraded nDCG@10 by -18.5% (BM25), -21.4% (LSA), and -20.2% (hybrid); all Holm-adjusted p-values are below 0.001. Cross-benchmark evaluation reveals that typo vulnerability scales steeply with document specificity: under identical typo perturbations, nDCG@10 degradation is moderate on technical abstracts (-2.5% on SciFact) but severe on medical terminology (-50.4% on NFCorpus). Paraphrasing reduced BM25 nDCG@10 by -13.3% (Holm p = 0.004), whereas changes for LSA (-5.3%) and the hybrid (-7.8%) were not statistically significant after correction. Natural-question rewrites and stop-word removal moved nDCG@10 by under 2% across all systems, while word-order permutations produced zero effect for these bag-of-words pipelines. Paraphrasing altered top-10 result order more than misspellings did (lower Rank-Biased Overlap) yet incurred lower metric loss, establishing that ranking displacement does not track effectiveness harm across variation categories. Complete source code, query variations, audit tools, and test suites are openly available.
Author: AHMED, MOHAMMED GHOUSE
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225566565