Data Versioning for Iterative Refinement: Adapting ML Experiment Tracking Tools for Data-Centric AI Pipelines
Year: 2024
Type: article
Abstract: The increasing emphasis on data-centric AI has highlighted the need for systematic approaches to manage evolving datasets in machine learning (ML) pipelines. While ML experiment tracking tools like ML flow and Weights & Biases (W&B) excel at versioning models and hyperparameters, they lack robust mechanisms for tracking dataset iterations such as corrections, augmentations, and subset selections that are critical in data-centric workflows. This paper bridges this gap by proposing a framework that extends existing ML experiment tracking paradigms to support data versioning, enabling reproducibility, auditability, and iterative refinement in data-centric AI. We draw inspiration from two key works: (1) "Dataset Versioning for Machine Learning: A Survey" (2023), which formalizes the challenges of dataset evolution tracking, and (2) "Data Fed: Towards Reproducible Deep Learning via Reliable Data Management" (2022), which introduces a federated data versioning system for large-scale ML. Our framework adapts these principles to integrate seamlessly with popular ML tracking tools, introducing data diffs (fine-grained change logs), provenance graphs (to track transformations), and conditional triggering (to automate pipeline stages based on data updates). We evaluate our approach on three real-world case studies: (a) a financial fraud detection system where transaction datasets are frequently revised, (b) a medical imaging pipeline with iterative label corrections, and (c) a recommendation engine with dynamic user feedback integration. Results show that our method reduces dataset reproducibility errors by 62% compared to ad-hoc versioning (e.g., manual CSV backups) while adding minimal overhead (<5% runtime penalty) to existing ML workflows. Additionally, we demonstrate how our framework enables data debugging by tracing model performance regressions to specific dataset changes a capability absent in current model-centric tools. This work contributes: (1) a methodology for adapting ML experiment trackers to handle dataset versioning, (2) an open-source implementation compatible with ML flow and W&B, and (3) empirical validation of its benefits across diverse domains. Our findings advocate for treating data as a first-class artifact in ML pipelines, aligning with the broader shift toward data-centric AI.
Author: Rajani Kumari Vaddepalli
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: bronze
OpenAlex ID: W4413488310