Work

Auto-Validate by-History: Auto-Program Data Quality Constraints to Validate Recurring Data Pipelines

Year: 2023

Type: conference-paper

Abstract: Data pipelines are widely employed in modern enterprises to power a variety of Machine-Learning (ML) and Business-Intelligence (BI) applications. Crucially, these pipelines are recurring (e.g., daily or hourly) in production settings to keep data updated so that ML models can be re-trained regularly, and BI dashboards refreshed frequently. However, data quality (DQ) issues can often creep into recurring pipelines because of upstream schema and data drift over time. As modern enterprises operate thousands of recurring pipelines, today data engineers have to spend substantial efforts to manually monitor and resolve DQ issues, as part of their DataOps and MLOps practices.

Source: ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

DOI: 10.1145/3580305.3599776

Authors: Dezhan Tu, Yeye He, Weiwei Cui, Song Ge, Haidong Zhang, Shi Han, Dongmei Zhang, Surajit Chaudhuri

Institutions: University of California, Los Angeles, Microsoft (United States), Microsoft Research Asia (China)

Language: English

Topic: Data Quality and Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

OpenAlex ID: W4385568252

FWCI: 3.973

Cites: 48

Cited by: 18

Related to: 10