Work

AcetylBench: a deduplicated benchmark dataset reveals systematic performance inflation and cross-database generalisation failure in lysine acetylation site prediction

Year: 2026

Type: article

Abstract: Lysine acetylation is a pervasive post-translational modification with critical regulatory roles, yet computational prediction of acetylation sites remains hampered by unreliable benchmarking practices including dataset redundancy and non-independent test set evaluation. We report a systematic investigation demonstrating that three independent sources of metric inflation, including near-ubiquitous sequence redundancy, non-independent test sets, and distribution-locked dimensionality reduction collectively reduced an apparent accuracy of 99.38% to a true cross-database AUC of 0.523, indistinguishable from random classification. Applying CD-HIT at 40% pairwise sequence identity to a dbPTM-derived acetylation dataset, we found that 62.7% of positive and 90.3% of negative sequences are redundant, a level not previously quantified for this task. A comprehensive cross-database benchmark evaluating seven representation strategies (CKSAAP, BLOSUM62, one-hot, ESM-2, ProtBERT, ESM-1v mean-pool, and ESM-1v lysine-position) across two training databases reveals a consistent generalisation failure: within-database cross-validation AUC values of 0.815–0.867 collapse to 0.582–0.607 on a truly independent benchmark. Logistic regression, random forest, and neural network classifiers achieve statistically indistinguishable cross-database performance on identical features, establishing that this failure is driven by feature representations rather than classifier architecture. DeepAcet, using PSSM features computed via PSI-BLAST against a database-independent evolutionary reference, achieves AUC 0.841, substantially exceeding all PLM-based methods trained on identical data (best: AUC 0.705). We release AcetylBench, a rigorously deduplicated benchmark comprising 27,182 positive and 7063 negative sequences, to enable reliable evaluation of future acetylation predictors.

Source: BMC Bioinformatics

DOI: 10.1186/s12859-026-06593-y

Authors: Muhammad Uzair Ashraf, Shaista Khan, Mohammad Sarfraz, Rizwan Hasan Khan

Institution: Aligarh Muslim University

Language: English

Topic: Machine Learning in Bioinformatics

Subfield: Molecular Biology

Field: Biochemistry, Genetics and Molecular Biology

Domain: Life Sciences

Open Access status: gold

Estimated APC paid: $2,990

Funder: Ministry of Education, India