Work

Cluster-based Partial Dense Retrieval Fused with Sparse Text Retrieval

Year: 2024

Type: conference-paper

Abstract: Previous work has demonstrated the potential to combine document rankings from dense and sparse retrievers for higher relevance effectiveness. This paper proposes a cluster-based partial dense retrieval scheme guided by sparse retrieval results to optimize fusion between dense and sparse retrieval at a low space and CPU-time cost while retaining a competitive relevance. This scheme exploits the overlap of sparse retrieval results and document embedding clusters, and judiciously selects a limited number of clusters to probabilistically guarantee the inclusion of top sparse results. This paper provides an evaluation of this scheme on its in-domain and zero-shot retrieval performance for the MS MARCO and BEIR datasets.

Source: International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

DOI: 10.1145/3626772.3657972

Authors: Yingrui Yang, Parker Carlson, Shanxiu He, Yifan Qiao, Tao Yang

Institution: University of California, Santa Barbara

Language: English

Topic: Information Retrieval and Search Behavior

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Universitas Brawijaya

Award: G2058114682