Work

Capturing end-to-end provenance for machine learning pipelines

Year: 2024

Type: article

Abstract: Modern workflows for developing ML pipelines utilize ML artifact management systems (ML AMSs) such as MLflow in addition to traditional version control systems such as Git. ML AMSs collect data, model, metadata and software artifacts used and produced in pipeline development workflows. While ensuring repeatability and reproducibility, the provenance capabilities are still rudimentary, mainly due to incomplete traces, coarse granularity, and limited query capabilities. In this paper, we introduce a comprehensive PROV-compliant provenance model that captures end-to-end provenance traces of ML pipelines, their artifacts, and their relationships based on MLflow and Git activities. Moreover, we present the tool MLflow2PROV for continuously extracting provenance graphs according to our model, enabling querying, analyzing, and processing of the collected provenance information.

Source: Information Systems

DOI: 10.1016/j.is.2024.102495

Authors: Marius Schlegel, Kai-Uwe Sattler

Institution: Technische Universität Ilmenau

Language: English

Topic: Scientific Computing and Data Management

Subfield: Information Systems and Management

Field: Decision Sciences

Domain: Social Sciences

Open Access status: hybrid

Estimated APC paid: $3,090

Funder: Thüringer Ministerium für Wirtschaft, Wissenschaft und Digitale Gesellschaft