Year: 2021
Type: article
Abstract: Many high-performance computing applications are of high consequence to society. Global climate modeling is a historic example of this. In 2020, the societal issue of greatest concern, the still-raging COVID-19 pandemic, saw a legion of computational scientists turning their endeavors to new research projects in this direction. Applications of such high consequence highlight the need for building trustworthy computational models. Emphasizing transparency and reproducibility has helped us build more trust in computational findings. In the context of supercomputing, however, we may ask: how do we trust results from computations that cannot be repeated? Access to supercomputers is limited, computing allocations are finite (and competitive), and machines are decommissioned after a few years. In this context, we might ask how reproducibility can be ensured, certified even, without exercising the original digital artifacts used to obtain new scientific results. This is often the situation in HPC. It is compounded now with greater adoption of machine learning techniques, which can be opaque. The ACM in 2017 issued a Statement on Algorithmic Transparency and Accountability, targeting algorithmic decision-making using data models \cite{council2017}. Among its seven principles, it calls for data provenance, auditability, validation and testing. These principles can be applied not only to data models, but to HPC in general. I want to discuss the next steps for reproducibility: how we may adapt our practice to achieve what I call unimpeachable provenance, and full auditability and accountability of scientific evidence produced via computation.An invited talk at SC20I was invited to speak at SC20 about my work and insights on transparency and reproducibility in the context of HPC. The session's theme was Responsible Application of HPC, and the title of my talk was "Trustworthy computational evidence through transparency and reproducibility." At the previous SC, I had the distinction to serve as Reproducibility Chair, leading an expansion of the initiative, which was placed under the Technical Program that year. We moved to make Artifact Description appendices required for all SC papers, created a template and an author kit for the preparation of the appendices, and introduced three new Technical Program tracks in support of the initiative. These are: the Artifact Description & Evaluation Appendices track—with an innovative double-open constructive review process—, the Reproducibility Challenge track, and the Journal Special Issue track, for managing the publication of select papers on the reproducibility benchmarks of the Student Cluster Competition. This year, the initiative was augmented to address issues of transparency, in addition to reproducibility, and a community sentiment study was launched to assess the impact of the effort, six-years in, and canvas the community's outlook on various aspects of it.Allow me to thank here Mike Heroux, Reproducibility Chair for SC in 2017 and 2018, Michela Taufer, SC19 General Chair—who put her trust in me to inherit the role from Mike—, and Beth Plale, the SC20 Transparency and Reproducibility Chair. I had countless inspiring and supportive conversations with Mike and Michela about the topic during the many months of planning for SC19, and more productive conversations with Beth during the transition to her leadership. Mike, Michela and I have served on other committees and working groups together, in particular, the group that met in July 2017 at the National Science Foundation (convened by Almadena Chtchelkanova) for the Workshop on Reproducibility Taxonomies for Computing and Computational Science. My presentation at that event condensed an inventory of uses of various terms like reproducibility and replication, across many fields of science \cite{barba2017}. I then wrote the review article "Terminologies for Reproducible Research," and posted it on arXiv \cite{barba2018}. It informed our workshop's report, which came out a few months later as a Sandia technical report \cite{taufer2018}. In it, we highlighted that the fields of computational and computing sciences provided two opposing definitions of the terms reproducible and replicable, representing an obstacle to progress in this sphere.The Association of Computing Machinery (ACM), representing computer science and industry professionals, had recently established a reproducibility initiative, and adopted diametrically opposite definitions to those used in computational sciences for more than two decades. In addition to raising awareness about the contradiction, we proposed a path to a compatible taxonomy. Compatibility is needed here because the computational sciences—astronomy, physics, epidemiology, biochemistry and others that use computing as a tool for discovery—and computing sciences (where algorithms, systems, software, and computers are the focus of study) have community overlap and often intersect in the venues of publication. The SC conference series is one example. Given the historical precedence and wider adoption of the definitions of reproducibility and replicability used in computational sciences, our Sandia report recommended that the ACM definitions be reversed. Several ACM-affiliated conferences were already using the artifact review and badging system (approved in 2016), so this was no modest suggestion. The report, however, was successful in raising awareness of the incompatible definitions, and the desirability of addressing it.A direct outcome of the Sandia report was a proposal to the National Information Standards Organization (NISO) for a Recommended Practice Toward a Compatible Taxonomy, Definitions, and Recognition Badging Scheme for Reproducibility in the Computational and Computing Sciences. NISO is accredited by the American National Standards Institute (ANSI) to develop, maintain, and publish consensus-based standards for information management. The organization has more than 70 members; publishers, information aggregators, libraries and other content providers use its standards. I co-chaired this particular working group, with Gerry Grenier from IEEE and Wayne Graves from ACM; Mike Heroux was also a member. The goal of the NISO Reproducibility Badging and Definitions Working group was to develop a Recommended Practice document—a step before development of a standard. As part of our joint work, we prepared a letter addressed to the ACM Publications Board, delivered in July 2019. It described the context and need for compatible reproducibility definitions and made the concrete request that ACM consider a change. By that time, not only did we have the Sandia report as justification, but the National Academies of Sciences, Engineering and Medicine (NASEM) had just released the report Reproducibility and Replicability in Science \cite{medicine2019}. It was the product of a long consensus study conducted by 15 experts, including myself, and sponsored by the National Science Foundation responding to Congressional decree. The NASEM report put forth its definitions as:Reproducibility is obtaining consistent results using the same input data, computational steps, methods and code, and conditions of analysis.Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.The key contradiction with the ACM badging system resides on which term comprises using the author-created digital artifacts (e.g., data and code). We stated in the NISO working-group letter that if the ACM definitions of reproducible and replicable could be interchanged, the working group could move forward towards its goal of drafting recommended practices for badging that would lead to wider adoption in other technical societies and publishers. The ACM Publications Board responded positively, and began working through the details on how to make changes to items already published in the Digital Library with the "Results Replicated" badge—about 188 items existed at that time that were affected. Over the Summer of 2020, the ACM applied changes to the published Artifact Review and Badging web pages, and added a version number. From version 1.0, we see a note added that, as a result of discussions with NISO, the ACM was harmonizing its terminologies with those used in the broader scientific research community.All this background serves to draw our attention to the prolonged, thoughtful, and sometimes arduous efforts that have been directed at charting paths for adoption and giving structure to reproducibility and replicability in our research communities. Let us move now to why and how might the HPC community move forward.Insights on transparent, reproducible HPC researchDeployed barely over a year ago, the NSF-funded Frontera system at the Texas Advanced Computing Center (TACC) came in as the 8th most powerful supercomputer in the world, and the fastest on a university campus. Up to 80% of the available time on the system is allocated through the NSF Petascale Computing Resource Allocation program. The latest round of Frontera allocations (as of this writing) was just announced on October 25, 2020. I read through the fact sheet on the 15 newly announced allocations, to get a sense for the types of projects in this portfolio. Four projects are machine-learning or AI-focused, the same number as those in astronomy and astrophysics, and one more than those in weather or climate modeling. Other projects are single instances spanning volcanology/mantle mechanics, molecular dynamics simulations of ion channels, quantum physics in materials science, and one engineering project in fluid-structure interactions. One could gather these HPC projects in four groups:Astronomy and astrophysics are mature fields that in general have high community expectations of openness and reproducibility. As I'll highlight below, however, even these communities with mature practices benefit from checks of reproducibility that uncover areas of improvement. The projects tackling weather and climate modeling are candidates for being considered of high consequence to society. One example from the Frontera allocations concerns the interaction of aerosols caused by industrial activity with clouds, which can end up composed of smaller droplets, and become more reflective, resulting in a cooling effect on climate. Global climate models tend to overestimate the radiative forcing, potentially underestimating global warming: why? This is a question of great consequence for science-informed policy, in a subject that is already under elevated scrutiny from the public. Another project in this cluster deals with real-time high-resolution ensemble forecasts of high-impact winter weather events. I submit that high standards of transparency, meticulous provenance capture, and investments of time and effort in reproducibility and quality assurance are justified in these projects. Four of the winning projects are applying techniques from machine learning to various areas of science. In one case, the researchers seek to bridge the gap in the trade-off between accuracy of prediction and model interpretability, to make ML more applicable in clinical and public health settings. This is clearly also an application of high consequence, but in addition all the projects in this subset face the particular transparency challenges of ML techniques, requiring new approaches to provenance capture and transparent reporting. The rest of the projects are classic high-performance computational science applications, such as materials science, geophysics, and fluid mechanics. Reproducible-research practices vary broadly in these settings, but I feel confident saying that all or nearly all those efforts would benefit from prospective data management, better software engineering, and more automated workflows. And their communities would grow stronger with more open sharing. The question I have is: how could the merit review of these projects nudge researchers towards greater transparency and reproducibility? Maybe that is a question for later, and a question to start with is how could support teams at cyberinfrastructure facilities work with researchers to facilitate their adoption of better practices in this vein? I'll revisit these questions later.I also looked at the 2019 Blue Waters Annual Report, released on September 15, 2020, with highlights from a multitude of research projects that benefitted from computing allocations on the system. Blue Waters went into full service in 2013 and has provided over 35 billion core-hour equivalents to researchers across the nation. The highlighted research projects fall into seven disciplinary categories, and include 32 projects in space science, 20 in geoscience, 45 in physics and engineering, and many more. I want to highlight just one out of the many dozens of projects featured in the Blue Waters Annual Report, for the following reason. I did a word search on the PDF with Zenodo, and that project was the only one listing Zenodo entries in the "Publications & Data Sets" section that ends each project feature. One other project (in the domain of astrophysics) mentions that data is available through the project website and in Zenodo, but doesn't list any data sets in the report. Zenodo is an open-access repository funded by the European Union's Framework Programs for Research, and operated by CERN. Some of the world’s top experts in running large-scale research data infrastructure are at CERN, and Zenodo is hosted on top of infrastructure built in service of what is the largest high-energy physics laboratory of the world. Zenodo hosts any kind of data, under any license type (including closed-access). It has become one of the most used archives for open sharing of research objects, including software.The project I want to highlight is "Molten-salt reactors and their fuel cycles," by at and and I many on computational science, including a to This project deals with modeling and of reactors and fuel physics and with the goal of of reactors in terms of and As part of the research by Blue the two software described as a for the and in a and a tool for fuel The in the project highlight include research in the of as as the Zenodo for and a publication about in the Journal of one of the of It is of that other projects of the Blue Waters have also made software archives in Zenodo or published their software in but did not it in this report and did not the the research context of the project I highlighted is of high The practices of this research group a high of transparency that be the in such transparency, the publication of the software in that it was subject to review and that it standards of the software, and on and leading to quality I want to highlight the work of a group that Michela and posted on arXiv The work to the that to the of using the data and that the had made available to the scientific The data had been by teams using leading to of the but no had been made at the original results. In this the report on challenges during the effort, even with of data and the original publication. was the of a single public repository with all the information needed to the The had the of one of the original had to that up being in the of in the of public Other highlights of the the original publication did not the version of the used in the the used to make the was not released one to it the original to data, which needed to be to with the public data In the the of the from a but not of the original as one is in The here is that even a that is mature in its standards of transparency and reproducibility checks to that these practices are or can be National Academies study on Reproducibility and Replicability in Science was by the National Science Foundation under Congressional with the from the Chair of the and NASEM and have a of on and and often the of NSF is in the of its to the report, and we can to more in the about and for researchers in the NASEM report are directed at all the various and and and in particular, to ask that applications discuss how assess and report and how the proposed work address reproducibility replicability It also that reproducibility and replicability in the of with for more transparency and public to the of we need to be of the more time, I have a to thank Mike Heroux, time for a call with me as I prepared my SC20 invited In as at of time is in service to the and this in the review of the & As it is an the for are not available but Mike me that now to include on how the project address transparency and reproducibility. These aspects are in the proposal review and are a in the As community expectations it could that between two in the science the from one of better addressing reproducibility. teams at Sandia are at a high an Artifact Description for publication of the conference or if or NSF might to general proposal researchers to transparency and reproducibility in the project One we see the start responding to expectations about open sharing of research is the section on results from NSF here a listing of from and of research and their including want to thank Beth Plale, time to with me over and me materials to use in my SC20 In 2020, NSF issued a on Science for with Beth then as the public The that NSF is its Access to leading to data and It research data to be in an service and a Digital a global and to the on the Data the to be and include any in the this be This that it later be The invited aimed at community to open science. At the same time, the of Science and issued a for Information this year what could do to make the results from research The on open science is and to the from the It other that to open sharing for data and code, with available for sharing of data and code, including and and open data and as of and and support for of data sharing and The various other like models for open and support for infrastructure needed to make open to research results more and It also that for and raising quality of data and code, and also and data such as for data, and for and machine data the National of just announced on October a new on data and sharing. It researchers to for managing and sharing scientific data to the of research to make data sharing and we could expectations to discuss reproducibility and open research is for of computing this I need to thank at the Texas Advanced Computing Center (and time for a call with me on this We about how providers might a role in adoption of reproducibility the NSF science proposal and the computing proposal are The Allocation discuss review which merit by the NSF science software, research and and use of the computational the most researchers have to that their application to the of the system are time the is not to and researchers are not to that their are only that and that the research question is to be in the allocated The of the system is provided for via a between the researchers and the of the are under the with of data (in the with and a web for see three system could be to also automated provenance the at the could their support to researchers to include and in transparency and reproducibility and facilities could their to include for reproducible other like the that projects be a on their allocations or to in more that projects may not be allocations over a and a of in steps for SC Transparency and Reproducibility is one of the efforts to the expectations and a community about how to address it and why it Over we have built community and This community sentiment study of the are of the issues reproducibility, and only the concerns are researchers report that are the artifact appendices of technical papers, As a we are better prepared to adapt to raising expectations from publishers, and has a of to and the is an example the COVID-19 and at is a global we as a are by public to data, code, and published results. This has been made by many in but most by and in their Data for by more than a researchers from the It is an need to our models, and results and to reproducibility, and it with to results available via to make available input and (e.g., and data to the to use open to make available results as as to and methods in to and and to these are of the I made in my Reproducibility the provided is a to in open science and to reproducibility. The question is how newly adopted practices the of a is here issues of for transparent and reproducible of science that sharing the results of are by and and a sharing of in a situation and In the of open sharing of research those expectations are not to to for one would like to see and scientists start sharing HPC settings, we can results to machine and a with the of transparency and reproducibility develop a of and consider the applications of high consequence to and support researchers with infrastructure and Over time, we at a of to achieve the goal of trustworthy computational not by exercising the open research by and but by a research that unimpeachable
Author: Lorena A. Barba
Institution: George Washington University
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $2,995
Award: G2450041348
OpenAlex ID: W3133755953