Year: 2026
Type: conference-paper
Abstract: Neural IR models achieve strong performance but remain difficult to interpret.We present NEAT-IR, a black-box analysis framework that explains ColBERT's ranking behavior using 26 classical IR features (BM25, TF-IDF, IDF measures, positional signals).We analyze Col-BERT through two complementary lenses: regression (predicting exact scores) and learningto-rank (predicting relative order), evaluated on MS MARCO (48,250 query-passage pairs).Our key finding is a score-rank gap: classical features preserve ColBERT's rankings nearly perfectly (N DCG@5 ≈ 0.99) yet explain only R 2 ≈ 0.28 of score variance.Feature attribution reveals that regression and ranking models rely on distinct feature subsets: query-level IDF signals dominate score prediction, while document-matching features (BM25, cosine TF-IDF) drive ranking preservation.These findings suggest that ColBERT's ordinal behavior on MS MARCO is largely recoverable from classical signals, while neural contributions primarily affect score magnitude.NEAT-IR enables practitioners to diagnose when neural rankers deviate from classical patterns, supporting interpretable model auditing and informed hybrid pipeline design.
Institution: Worcester Polytechnic Institute
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Worcester Polytechnic Institute
OpenAlex ID: W7166805891