Work

PathwayBench: a multi-criterion benchmark of pseudobulk pathway activity scoring methods reveals rank-window truncation as a two-sided mechanism of biological signal loss in single-cell RNA-seq

Year: 2026

Type: article

Abstract: Pathway activity scoring is a widely used step in single-cell RNA-seq (scRNA-seq) analysis, yet where it is applied, method choice is rarely guided by systematic, multi-criterion evidence in the pseudobulk case-control regime that now dominates applied single-cell disease studies. Prior single-cell pathway-scoring benchmarks have focused on perturbation ground truth at cell-level resolution, leaving the donor-level pseudobulk setting—and the question of why methods disagree—largely unaddressed. We present PathwayBench, a benchmark comparing five widely used pathway scoring methods (ssGSEA, GSVA, z-score, AUCell, UCell) across eight pseudobulked scRNA-seq case-control datasets spanning five tissues (brain, heart, kidney, lung, blood) and 682 donors. Methods are evaluated against five criteria covering biological relevance (direction accuracy, AUROC, effect size) and four robustness axes (aggregation, outlier, normalization, sample-size stability). We identify rank-window truncation as a two-sided mechanism by which top-window rank methods (AUCell, UCell) lose biological signal: they go blind when pathway genes fall outside the fixed top-rank window, and they attenuate or invert when non-pathway competitor genes invade that window. A 49-condition simulation sweep shows the effect is broad across the parameter grid: AUCell returns a null score in 33 of 49 conditions when pathway genes fall below its default window, and both rank methods invert sign under heavy competitor burden. A window-parameter experiment on real data shows that widening the rank window recovers most of the lost signal at an intermediate setting, confirming that the default single-cell window is mis-set for dense pseudobulk profiles. Real-data illustration comes from extracellular matrix (ECM) remodeling in chronic kidney disease, where top-window rank methods produce near-zero or wrong-direction effects across three normalizations. Discretizing per-criterion performance as Good/Intermediate/Poor indicates that no single method satisfies all five criteria under the primary thresholds. Because this classification is sensitive to threshold and weighting choices, we report the underlying continuous scores as the primary evidence and use the discrete scoreboard only as a summary aid. Rank-window truncation provides a mechanistic explanation for systematic divergence between whole-distribution methods and top-window truncated-rank methods on scRNA-seq data. We adopt a three-family taxonomy—cross-sample standardized magnitude (z-score), rank-enrichment hybrids (ssGSEA, GSVA), and top-window truncated rank (AUCell, UCell)—which is more accurate than a two-way magnitude-versus-rank split, since ssGSEA is itself a rank-based enrichment statistic and should guide method selection, particularly for fibrotic, inflammatory, or otherwise broadly remodeled transcriptomes. PathwayBench is delivered as a versioned, extensible benchmark with per-dataset scores, an interactive advisor application, and complete reproducibility infrastructure to support evidence-based method selection and community extension.

Source: BMC Bioinformatics

DOI: 10.1186/s12859-026-06632-8

Authors: Fadhl M. Alakwaa, Ahmed Elbaz, Sara Al Azzam, Tasnim Alshebani

Institutions: University of Michigan, Michigan Medicine, American University in Cairo, Jordan University of Science and Technology, Wayne State University

Language: English

Topic: Bioinformatics and Genomic Networks

Subfield: Molecular Biology

Field: Biochemistry, Genetics and Molecular Biology

Domain: Life Sciences

SDG: Good health and well-being

Open Access status: gold

Estimated APC paid: $2,990