Integrating advanced compression techniques with key-value stores for managing large source-code datasets
Year: 2024
Type: dissertation
Abstract: Large source-code datasets are rising in popularity for archival and artificial intelligence purposes. We find new ways to optimize the storage and indexing of these datasets, achieving better compression ratios and access throughput to the files. To this aim, we leverage the Permute-Partition-Compress paradigm and the RocksDB key-value store, in order to make the datasets compressible, dynamic and quickly accessible. We corroborate the practical efficiency of our solution by performing experiments on datasets of various sizes up to 6 TB, taken from The Stack v1 (provided by HuggingFace) and the Software Heritage Archive. Datasets and source code are made publicly available on Github.
Author: RAMACCIOTTI, FEDERICO
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7057326213
Cites:
Cited by:
Related to: