Year: 2025
Type: article
Abstract: In modern data ecosystems, ensuring integrity, traceability, and auditability of data is critical to building trust and enabling compliance. This article explores practical strategies for implementing data versioning and lineage tracking within ETL (Extract, Transform, Load) workflows. It outlines versioning techniques such as snapshotting and Change Data Capture (CDC), as well as lineage tracking methods including direct capture and log - based inference. The paper also discusses how to leverage tools like Delta Lake, Apache Atlas, Open Lineage, and Neo4j to manage metadata and visualize data flow. Through real - world examples and implementation guidance, readers will learn how to design resilient, transparent ETL pipelines that support robust data governance and operational efficiency.
Authors: Mounica Achanta, Dharanidhar Vuppu
Institution: SurveyMonkey (United States)
Language: English
Topic: Data Quality and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W4410747361