Self-supervised data lakes discovery through unsupervised metadata-driven weighted similarity
Year: 2024
Type: article
Abstract: Data engineers invest significant effort in the early stages of data analysis, including identifying relevant datasets in large and complex data lakes. Data retrieval efficiency becomes an urgent concern as data volumes continue to increase with structures that vary in size and complexity. This paper presents a novel strategy for accelerating data discovery in data lakes. Our approach integrates self-supervised techniques and weighted similarity estimation for efficient dataset classification, facilitating faster search and retrieval. By extracting meta-feature characteristics, our approach improves data traceability in data lakes through clustering, resulting in significant improvements in modularity, showing an efficiency gain of 69.8%. Regarding dataset search through classification, it consistently achieves AUC-ROC scores exceeding 0.85, indicating strong performance in class differentiation. Finally, our method reduces the overall execution time more than twofold and shows promising applications for addressing real-world challenges in various data lake domains.
Source: Information Sciences
Authors: I Made Putrama, Peter Martinek
Language: English
Topic: Data Quality and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $3,360
OpenAlex ID: W4391428836