Year: 2025
Type: article
Abstract: The position of data engineers has been advanced substantially over the last ten years with the evolution of Extract, Transform, Load (ETL) processes, Master Data Management (MDM), and cloud data platforms. In spite of the quicker uptake of cloud technology and automating ETL processes, numerous gaps remain in managing, processing, and quality checking large amounts of data efficiently in the modern enterprise landscape. Past research indicates there is a requirement for more efficient ways to integrate heterogeneous data sources, resolve data quality problems, and facilitate smooth interaction amongst data engineering teams in decentralized platforms. Although cloud-based data warehousing solutions and ETL tools have provided better scalability, security, and cost advantages, they are still afflicted with issues of real-time data integration, data consistency, and governance in multiple cloud environments. Further, existing Master Data Management (MDM) techniques are not able to guarantee data consistency and integrity in dynamic, distributed environments. Applying Artificial Intelligence (AI) and Machine Learning (ML) to automate the monitoring of data quality, detect anomalies, and applying self-healing mechanisms is an emerging topic yet to be explored in detail and implemented as part of standard data engineering processes.
Author: Kishore Ande
Institution: CVS Health (United States)
Language: English
Topic: Data Quality and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4409350397