Year: 2024
Type: dissertation
Abstract: Companies and organizations depend heavily on their data to make informed business decisions.Therefore, guaranteeing high data quality is critical to ensure the reliability of data analysis.Data integration, which aims to combine data acquired from several heterogeneous sources to provide users with a unified consistent view, plays a fundamental role to enhance the value of the data at hand.When data integration involves a limited number of sources, ETL (Extract, Transform, Load) is generally adopted as a paradigm: raw data is collected, cleaned, and stored in a data warehouse to perform analysis on it.Nowadays, big data integration needs to deal with millions of sources; thus, the paradigm is more and more moving towards ELT (Extract, Load, Transform).A huge amount of raw data is collected and directly stored (e.g., in a data lake), then different users can transform portions of it according to the task at hand.Hence, novel approaches to data integration need to be explored to address the challenges raised by this paradigm.One of the fundamental building blocks for data integration is Entity Resolution (ER), which aims at detecting records that describe the same realworld entity, to consolidate them into a single consistent representation.ER is typically employed as an expensive offline cleaning step on the entire data before consuming it.Yet, determining which entities are useful once cleaned depends solely on the user's application, which may need only a fraction of them.For instance, when dealing with Web data, we would like to be able to filter the entities of interest gathered from multiple sources without cleaning the entire continuously growing data.Similarly, when querying data lakes, we want to transform data on-demand and return results in a timely manner.Hence, we propose BrewER, a solution to evaluate SQL SP queries on dirty data while progressively returning results as if they were issued on the cleaned data.BrewER tries to focus the cleaning effort on one entity at a time, according to the priority defined by the user through the ORDER BY clause.For a wide range of applications (e.g., data exploration), a significant amount of resources can therefore be saved.Further, duplicates not only exist at record level, as in the case for ER, i ii but also at dataset level.In the ELT scenario, it is common for data scientists to retrieve datasets from the enterprise's data lake, perform transformations for their analysis, then store back the new datasets into the data lake.Similarly, in Web contexts such as Wikipedia, a table can be duplicated at a given time, with the different copies having independent development, possibly leading to the insurgence of inconsistencies.Automatically detecting duplicate tables would allow to guarantee their consistency through data cleaning or change propagation, but also to eliminate redundancy to free up storage space or to save additional work for the editors.While dataset discovery research developed efficient tools to retrieve unionable or joinable tables, the problem of detecting duplicate tables has been mostly overlooked in the existing literature.To fill this gap, we therefore present Sloth, a solution to efficiently determine the largest overlap (i.e., the largest common subtable) between two tables.The detection of the largest overlap allows to quantify the similarity between the two tables and spot their inconsistencies.BrewER and Sloth represent novel solutions to perform big data integration in the ELT scenario, fostering on-demand use of available resources and shifting this fundamental process towards a task-driven paradigm.
Authors: Bergamaschi, Sonia, Simonini, Giovanni, Rovati, Luigi, Zecchini, Luca
Institution: University of Modena and Reggio Emilia
Language: Italian
Topic: Data Quality and Management
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7020127289
