Work

Powerful Significance Testing for Unbalanced Clusters

Year: 2025

Type: article

Abstract: Clustering methods are popular for revealing structure in data, particularly in the high-dimensional setting common to contemporary data science. A central statistical question is “are the clusters really there?” One pioneering method in statistical cluster validation is SigClust, but it is severely underpowered in the important setting where the candidate clusters have unbalanced sizes, such as in rare subtypes of disease. We show why this is the case and propose a remedy that is powerful in both the unbalanced and balanced settings, using a novel generalization of k-means clustering. We illustrate the value of our method using a high-dimensional dataset of gene expression in kidney cancer patients. A Python implementation is available at https://github.com/thomaskeefe/sigclust. Supplementary materials for this article are available online.

Source: Journal of Computational and Graphical Statistics

DOI: 10.1080/10618600.2025.2469756

Authors: Thomas H. Keefe, J. S. Marron

Institution: University of North Carolina at Chapel Hill

Language: English

Topic: Advanced Clustering Algorithms Research

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: National Institute of Arthritis and Musculoskeletal and Skin Diseases

Award: G513288951