Work

RabbitCD-HIT: Distributed High-Sensitivity Greedy Incremental Clustering for Billion-Scale Protein Sequences in Hours

Year: 2026

Type: preprint

Abstract: We present RabbitCD-HIT, a scalable distributed high-sensitivity greedy incremental clustering tool for ultra-large protein sequence datasets. It fully exploits cluster resources to enable efficient billion-scale clustering. RabbitCD-HIT clustered the March 10, 2026 NCBI nr snapshot (1.02 billion sequences; 487 GB FASTA) at 90\% identity within 5 hours on 64 nodes; the resulting representatives and cluster assignments are released as a public database resource. On a 256-core server, RabbitCD-HIT achieved a 2.1$\times$--6.6$\times$ speedup over CD-HIT with identical results under matched settings. Our approach addresses performance bottlenecks of traditional greedy incremental algorithms and provides an effective solution for large-scale protein sequence clustering.

Source: bioRxiv (Cold Spring Harbor Laboratory)

DOI: 10.64898/2026.11.05.764756

Authors: Yijie Gao, Guiliang Ma, Zekun Yin, Lifeng Yan, Yang Yang, Tong Zhang, Tiancheng Zhao, Zhuo Liu, Xiaohui Duan, Beifang Niu, Bertil Schmibt, Weiguo Liu

Institutions: Shandong University, Georgia Institute of Technology, Chinese Academy of Sciences, Computer Network Information Center, Johannes Gutenberg University Mainz

Language: English

Topic: Genomics and Phylogenetic Studies

Subfield: Molecular Biology

Field: Biochemistry, Genetics and Molecular Biology

Domain: Life Sciences

Open Access status: green

OpenAlex ID: W7225630796

Cites:

Cited by:

Related to: