Work

Random Splits Overstate Antimicrobial Resistance Prediction: A Mechanism-Stratified Benchmark in E. coli and K. pneumoniae

Year: 2026
Type: dataset
Abstract: This repository contains the complete code, intermediate results, trained models, and figures accompanying the manuscript "Random Splits Overstate Antimicrobial Resistance Prediction: A Mechanism-Stratified Benchmark in E. coli and K. pneumoniae". The study benchmarks machine learning models (logistic regression, random forest, and XGBoost) predicting antimicrobial resistance from gene presence/absence data across ten species-antibiotic combinations in Escherichia coli and Klebsiella pneumoniae, drawn from the EMBL-EBI AMR Portal. Rather than asking whether machine learning can predict resistance, the work asks under what evaluation conditions any observed advantage over existing genotype-to-phenotype approaches actually holds. Three evaluation strategies are compared head-to-head for every combination: a conventional random split, a genomic-similarity-aware split (isolates clustered by gene-content Jaccard distance and held out by whole cluster, as a proxy for clonal relatedness), and a geographic leave-region-out split. Model performance under each is benchmarked against a deterministic, gene-presence-based rule comparator, and every model's SHAP feature attributions are cross-validated against the CARD (Comprehensive Antibiotic Resistance Database) to check whether the genes driving predictions are ones with documented resistance mechanisms. Key findings reproducible from this repository: random-split evaluation overstates genuine generalisation by a mean of 0.029 AUC-ROC (95% CI [0.023, 0.035]) relative to the genomic-similarity-aware split; the machine-learning advantage over a simple gene-presence rule is mechanism-dependent, negligible for some antibiotic-resistance mechanisms and large and rigor-stable for others (e.g. fluoroquinolone resistance in K. pneumoniae, where the causal mutations are not representable by gene presence/absence alone); and 66.0% of the top SHAP-ranked genes across all ten combinations have a confident, organism-aware match in CARD. The repository is organized as a runnable pipeline (twelve numbered Python scripts covering data profiling, bias-corrected genotype-phenotype linkage, dataset construction, the three splitting strategies, model training, the rule-based comparator, bootstrap statistical testing, SHAP/CARD validation, hyperparameter sensitivity checks, and figure generation), together with the raw and processed data, the final trained model weights for every combination, and every numeric result and figure reported in the manuscript. Code is generic and dependency-documented (requirements.txt); no cloud-notebook- or vendor-specific setup is required to reproduce any stage.
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7207828225
Cites:
Cited by:
Related to: 10