Work

Reproducibility Requires Consolidated Artifacts

Year: 2023

Type: conference-paper

Abstract: Machine learning is facing a ‘reproducibility crisis’ where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.

Source: IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN)

DOI: 10.1109/cain58948.2023.00025

Authors: Iordanis Fostiropoulos, Bowman Brown, Laurent Itti

Institution: University of Southern California

Language: English

Topic: Scientific Computing and Data Management

Subfield: Information Systems and Management

Field: Decision Sciences

Domain: Social Sciences

Open Access status: green

Funders: Semiconductor Research Corporation, Defense Advanced Research Projects Agency, Army Research Office

Awards: G3879893252, G486694964