Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks
Year: 2021
Type: article
Abstract: Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.
Source: Nature Communications
Authors: Julia Koehler Leman, Sergey Lyskov, Steven M. Lewis, Jared Adolf-Bryfogle, Rebecca F. Alford, Kyle Barlow, Ziv Ben-Aharon, Daniel Farrell, Jason Fell, William A. Hansen, Ameya Harmalkar, Jeliazko Jeliazkov, Georg Kuenze, Justyna D. Krys, Ajasja Ljubetič, Amanda L. Loshbaugh, Jack Maguire, Rocco Moretti, Vikram Khipple Mulligan, Morgan L. Nance, Phuong T. Nguyen, Shane Ó Conchúir, Shourya S. Roy Burman, Rituparna Samanta, Shannon T. Smith, Frank Teets, Johanna K. S. Tiemann, Andrew Watkins, Hope Woods, Brahm J. Yachnin, Christopher D. Bahl, Chris Bailey-Kellogg, David Baker, Rhiju Das, Frank DiMaio, Sagar D. Khare, Tanja Kortemme, Jason W. Labonte, Kresten Lindorff-Larsen, Jens Meiler, William Schief, Ora Schueler-Furman, Justin B. Siegel, Amelie Stein, Vladimir Yarov-Yarovoy, Brian Kuhlman, Andrew Leaver-Fay, Dominik Gront, Jeffrey J. Gray, Richard Bonneau
Institutions: Simons Foundation, Flatiron Institute, New York University, Johns Hopkins University, Cyrus Biotechnology, International AIDS Vaccine Initiative, University of California, San Francisco, Hebrew University of Jerusalem, University of Washington, University of California, Davis, Rutgers, The State University of New Jersey, Vanderbilt University, Leipzig University, University of Warsaw, University of North Carolina at Chapel Hill, University of Copenhagen, Stanford Medicine, Stanford University, Boston Children's Hospital, Harvard University, Institute for Protein Innovation, Dartmouth College
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $5,560
Funders: National Science Foundation, Howard Hughes Medical Institute, Cancer Research Institute, Simons Foundation, American Heart Association, Deutsche Forschungsgemeinschaft, Lundbeckfonden, Israel Science Foundation, Novo Nordisk, Novo Nordisk Fonden, H. Lundbeck A/S, National Institutes of Health, Horizon 2020 Framework Programme, Multidisciplinary University Research Initiative, National Heart, Lung, and Blood Institute, National Institute on Drug Abuse, National Cancer Institute, National Institute of General Medical Sciences, National Institute of Allergy and Infectious Diseases, National Institute of Neurological Disorders and Stroke, National Institute of Environmental Health Sciences, National Institute of Biomedical Imaging and Bioengineering, Division of Chemical, Bioengineering, Environmental, and Transport Systems, Division of Biological Infrastructure, Army Research Office
Awards: G1049944735, G1064193810, High-Performance Computer Cluster for Biomedical Research, G1232145490, G1308682533, G1329933091, G1348053060, G1445918810, G146225979, Technologies to predict and probe glycosyl transfer, Hybridized structure- and ligand- based drug discovery approaches targeting ASCT2, an amino acid transporter critical for upregulated cell proliferation in numerous cancer types, Collaborative Research: Understanding and exploiting the structure-function link between fatty acid biosynthesis and degradation enzymes for functionalized small molecule synthesis, Computational de novo design of a disulfide-rich miniprotein synthetic library and its application to engineer binders to neutralizing epitopes on Clostridium difficile toxins TcdA and TcdB, Molecular Biophysics Training Grant, Biomarkers of Exposure to Hazardous Substances, Rosetta: An Integrated Macromolecular Modeling Suite, In silico safety pharmacology, G2358722041, G2401209729, G2421906508, G2705043322, Structure based design of trimer interface epitope focused universal influenza vaccines, G2830696884, Hybrid Methods for Prediction and Design of Novel Human Influenza Antibodies, Functional Deimmunization of Therapeutic Proteins, G3611247453, G3751405579, De novo design of generalizable allosteric modulators and peptide ligands for G protein coupled receptors, G3874321780, Predoctoral Training Program in Bioinformatics and Computational Biology, Membrane Protein Structure Elucidation from Limited Magnetic Resonance Data, Optimization of non-addictive biologics to target sodium channels involved in pain signaling, Computational tools for understanding chemically modified RNA structure and interactions, Structural mechanism of polymodal TRP channel activation, Computer Modeling of the Potassium Channel KCNQ1 to Study Protein Function and the Role of Mutations in Heart Arrhythmia, G5462754644, G5520316217, G5562136253, RAPID: Arresting the spread of SARS-CoV-2 on surfaces and in the air using engineered water nanostructures enriched with de novo designed neutralizing peptides, G5734038057, G5941404119, RCN-UBE: Data-to-Design Course-based Undergraduate Research Experience ? protein modeling and characterization to enhance student learning and improve computational protein design, G606658003, Topological Energetics and the Cellular Quality Control of Integral Membrane Proteins, G657448715, ABI Innovation: Robotics-inspired modeling & design of proteins, CI-EN: Collaborative Research: Enhancement of Foldit, a Community Infrastructure Supporting Research on Knowledge Discovery Via Crowdsourcing in Computational Biology, Structure based design of antibodies and vaccines, Immunology Training, G6864327681, Principles for Designing Stimulus-Responsive Enzymes, G7452299184, G7493659275, Biosynthesis of (di)terpene scaffolds, Computational Modeling of MUC1 Glycoforms for the Development of Cancer Therapeut, Prediction of the Structure of Therapeutic Antibodies with their Antigens, G7697847498, Computational Design of Crystal Lattice Interactions to Determine Recalcitrant Protein Structures, Structural Determinants of Allosteric Modulation of Brain GPCRs, Activation and desensitization of heat-sensor TRPV1, Directed Biomineralization: Designing Peptides to Control Crystal Nucleation and Growth, G8245670873, Glycan dependent epitopes of HIV broadly neutralizing antibodies, G8278764384, G8538192389, G8581368535, G8593986513, Computational design of new protein structures and interactions, High-resolution modeling of protein-RNA interfaces, G8858395165, G9000353076, Modeling and design of complex RNA structures, Computational Design of Protein Structures and Complexes, Protein structure determination from low-resolution experimental data
OpenAlex ID: W3215748819