Design and Implementation of Data Quality Optimization Strategy and Multi-Level Data Governance Platform based on Big Data Processing and Dynamic Rule Extension
Year: 2025
Type: conference-paper
Abstract: With the rapid evolution of Internet, cloud computing and Internet of Things technology, society has entered a new era based on big data. In this context, with the help of advanced data processing technology, the big data credit reporting industry has changed the way of data collection, processing and analysis, so that multi-dimensional and hierarchical data can be effectively used for credit assessment and continue to tap its potential value. However, the widespread use of massive data also raises a series of challenges. Data integration is difficult because different organizations have different data formats, and existing synchronization tools need to be improved in terms of versatility and real-time updates. At the same time, the problem of data traceability is prominent, and the accuracy of traditional blood analysis methods is insufficient, which leads to the complicated and difficult process of extracting blood information. Data quality is also worrying, the record is arbitrary, and the data from logs and texts lacks integrity and standardization. In order to deal with these problems, this paper focuses on the key technologies of data governance, and designs and implements a comprehensive governance platform for big data credit investigation. The core of the research is to build an operation method that supports offline and real-time data synchronization, and improve the integration effect of multi-source data by optimizing the configuration process. Develop a lineage analysis technique for Flink SQL to reduce the impact of high coupling and low accuracy, and ensure the accuracy and low intrusion of the parsing process. In addition, the construction of a multi-level data governance system aims to improve the quality of credit information by effectively integrating and synchronizing various data, so as to provide strong data support for the credit information business of individuals and enterprises. After verification, the designed system shows excellent performance, laying a solid foundation for the future development of the field of big data credit investigation.
Source: Procedia Computer Science
Author: Fengyi Zhao
Institution: Intercontinental Exchange (United States)
Language: English
Topic: Data Quality and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W4411175204