Entropy-Minimized Byte-Plane Decomposition: Auto-Detection and the Universality Gap in Lossless Compression
Year: 2026
Type: preprint
Abstract: Byte-plane decomposition — reordering serialized multi-byte data by byte position before compression — is widely deployed in scientific data pipelines (HDF5, Blosc, Apache Parquet) but requires the user or application schema to specify the field width. We present two schema-free algorithms for automatically detecting the optimal decomposition width by minimizing average per-plane Shannon entropy: one over a fixed candidate set, and one using sampled byte-match frequency to detect arbitrary widths without a predefined set. The algorithms are validated on data with record widths from 2 to 36 bytes, including the SAO Star Catalog where width=28 is correctly identified while all power-of-two widths fail. We additionally provide the first systematic characterization of the universality gap between structure-aware and structure-blind lossless compression. Across five datasets — synthetic sensor telemetry, real-world IoT sensors, MRI medical imaging, astronomical catalog data, and financial order book data — the gap ranges from 1.00× to 40.8× and correlates with the MSB-to-LSB entropy gradient and absolute MSB entropy within multi-byte fields. All results are validated on the BPD implementation, a block-independent ZSTD-backed compression platform implementing schema-free width detection and per-block strategy selection. Code: https://github.com/leeroy37/byte-plane-decomposition
Author: Goehring, Steffen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7202389753
Cites:
Cited by:
Related to: