Calibration Quality and Query-Level Risk Detection in BM25 Retrieval
Year: 2026
Type: article
Abstract: Internet-facing search and retrieval-augmented generation systems often use retrieval scores as confidence signals to route, filter, or withhold results, although ranking quality and probabilistic confidence are different properties. We test whether better-calibrated BM25 scores are more useful for such query-level decisions, using MS MARCO passages with TREC Deep Learning 2019 and 2020 judgments. Naive min–max normalization, Platt scaling, and isotonic regression are compared in a two-fold cross-collection design, and detection of below-median retrieval is evaluated with a fold-stratified area under the receiver operating characteristic curve (AUROC). Naive normalization was not calibrated across collections, and calibrating raw scores did not outperform a constant predictor at the training-set prevalence (pooled Brier score 0.211 for both). Normalizing before calibration gave the lowest Brier scores (0.201–0.202) and the lowest miscalibration in point estimate. The estimators with lower point-estimate calibration error did not demonstrate better risk detection: they reached AUROCs of 0.562–0.568, close to normalization alone (0.562) and below a top-100 score-dispersion baseline (0.731). This held for top-100 calibrated aggregates, absolute effectiveness thresholds, and calibrators refitted in the bootstrap, although moderate gains or losses cannot be excluded. Calibration quality and decision utility should be validated separately before retrieval scores serve as control signals.
Source: Future Internet
DOI: 10.3390/fi18110639
Author: Khaled Albishre
Institution: Umm al-Qura University
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,930
Funder: Umm Al-Qura University
Award: G9001399407
OpenAlex ID: W7236735693
