Get started
Work

MiCAS: Mixture-based Cell Annotation with Open-Set Recognition for scRNA-seq Data

Year: 2026
Type: preprint
Abstract: Background Single-cell RNA sequencing (scRNA-seq) enables detailed characterization of cellular heterogeneity, but accurate cell type annotation remains an important challenge in single-cell analysis. Most existing methods operate under a closed-set assumption and assign every query cell to one of the predefined reference cell types. However, real biological datasets may contain rare, novel, or previously unseen cell populations, which may lead to incorrect and overconfident predictions. To address this limitation, we propose MiCAS (Mixture-based Cell Annotation with Open-Set Recognition), an open-set framework that represents known cell populations using Gaussian Mixture Models optimized via Particle Swarm Optimization, and applies class-specific rejection thresholds calibrated through ROC analysis to identify low-confidence cells as unknown. Results The proposed framework was evaluated on four publicly available scRNA-seq datasets under controlled open-set conditions and compared with commonly used annotation methods. MiCAS achieved the highest AUROC across all datasets (0.866–0.925 vs. 0.62–0.85 for competing methods) and open-set accuracy (\(\:{A}_{o}\)) of up to 0.80. It successfully detects previously unseen cell populations while maintaining competitive annotation performance for known cell types. Conclusions MiCAS provides a probabilistic framework for open-set cell type annotation in scRNA-seq data. The proposed approach improves the reliability of cell type annotation under realistic biological conditions. The framework provides a practical foundation for future open-set single-cell analysis, particularly in studies involving heterogeneous tissues and the discovery of rare or novel cell populations.
DOI: 10.21203/rs.3.rs-10438566/v1
Institution: Inonu University
Language: English
Open Access status: green
OpenAlex ID: W7214403427
Cites: 29
Cited by:
Related to: 10