Year: 2024
Type: dissertation
Abstract: Technology developments have rapidly changed manufacturing in recent years making data a paramount asset. Data pipelines are vital in collecting, transferring, and processing data for ever-increasing business-critical applications. Data quality also plays a crucial role in the effectiveness and efficiency of decision-making processes within organizations, particularly in the manufacturing sector. Companies still face challenges with both hindering the potential of data. The thesis aims to identify common data quality challenges in a global manufacturing company and examine their possible causes. Additionally, it explores the design and development of an automated monitoring solution to address potential data incompleteness issues with the existing data pipeline. \n \nThe thesis study is conducted as design science research and consists of a literature review, expert interviews, and empirical research describing the design process. The literature review mainly examines data pipelines, quality, and anomaly detection. The empirical research investigated common data quality problems and the development process for implementing automated data pipeline monitoring. The development process investigates multiple solution possibilities and applicable designs to monitor data loss and completeness of records. The study also composes a general framework deriving knowledge from literature and empirical research. \n \nThe study concludes that common data quality problems in the company are outdated data, discrepancies between source systems, missing data, and human-induced errors. The company's probable cause for these data quality problems is lacking data governance. The thesis study develops a customized automated monitoring solution that fits into the existing data pipeline of the company increasing visibility and trust. This solution involves using data from logging mechanisms to establish real-time monitoring, error analysis, and alerting mechanisms notifying relevant stakeholders about potential data loss or anomalies in data pipeline performance. Also, the study introduces a general framework for developing data monitoring to an existing pipeline contributing to the knowledge base. In the future, the general framework can be used by the company and other practitioners to guide pipeline monitoring.
Source: LUTPub (LUT University)
Author: Mattila, Riku
Institution: Lappeenranta-Lahti University of Technology
Language: English
Topic: Data Quality and Management
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7055635142
