Integrating Unsupervised Clustering with Transformer-Based Models for De Novo Cell Population Identification
Year: 2026
Type: preprint
Abstract: Single-cell transcriptome sequencing has been widely applied across the life sciences. It enables accurate and robust identification of cell types in heterogeneous biological tissues through unsupervised clustering, shallow learning, and deep learning. While deep learning does not outperform shallow learning in classification performance, it streamlines cell type identification by eliminating cumbersome preprocessing procedures. However, the availability of training datasets with comprehensive cell type annotations is always limited, so new datasets inevitably contain cell populations that are not pre-defined in the training data. To address this limitation, we integrated unsupervised clustering with transformer-based models to identify cell populations unrecognized by classical methods. Specifically, we developed the Nankai GeneRank Classifier by fine-tuning the pre-trained Geneformer-V1 model, and then clustered cells based on its softmax-normalized class probability matrix (SCPM). Applied to a public colorectal cancer dataset, the new method, named SCPM-based clustering, resolved cell populations that cannot be unambiguously identified by classical methods alone and led to the discovery of novel candidate markers of colorectal cancer stem cells, demonstrating its utility.
Institutions: Yunnan University, Nankai University, Qilu Normal University
Language: English
Subfield: Molecular Biology
Domain: Life Sciences
Open Access status: green
Funder: Nankai University
OpenAlex ID: W7234027165
