Year: 2024
Type: conference-paper
Abstract: The k-Minimum Values (KMV) data sketch algorithm stores the k least hash keys generated by hashing the items in a dataset. We show that compression based on ordering the keys and encoding successive differences can offer O(log n) bits per key in expected storage savings, where n is the number of unique values in the data set. We also show that O(log n) expected bits saved per key is optimal for any form of compression for the k least of n random values – that the encoding method is near-optimal among all methods to encode a KMVsketch.We present a practical method to perform that compression, show that it is computationally efficient, and demonstrate that its average savings in practice is within about five percent of the theoretical minimum based on entropy. We verify that our method outperforms off-the-shelf compression methods, and we demonstrate that it is practical, using real and synthetic data.1
Authors: Charlie Dickens, Eric Bax, Alexander Saydakov
Institution: Yahoo (United States)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4406495605