Year: 2022
Type: preprint
Abstract: Lossy compression is one of the most efficient solutions to reduce storage overhead and improve I/O performance for HPC applications. However, existing parallel I/O libraries cannot fully utilize lossy compression to accelerate parallel write due to the lack of deep understanding on compression-write performance. To this end, we propose to deeply integrate predictive lossy compression with HDF5 to significantly improve the parallel-write performance. Specifically, we propose analytical models to predict the time of compression and parallel write before the actual compression to enable compression-write overlapping. We also introduce an extra space in the process to handle possible data overflows resulting from prediction uncertainty in compression ratios. Moreover, we propose an optimization to reorder the compression tasks to increase the overlapping efficiency. Experiments with up to 4,096 cores from Summit show that our solution improves the write performance by up to 4.5X and 2.9X over the non-compression and lossy compression solutions, respectively, with only 1.5% storage overhead (compared to original data) on two real-world HPC applications.
Source: arXiv (Cornell University)
Institutions: Washington State University, Argonne National Laboratory
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G102163711, G1071216263, G1193633358, G1286236842, G1313983767, G1664757510, G1796307089, G1799333409, G1863003419, G2287611682, G262962599, G2638771445, CDS&E: Collaborative Research: HyLoC: Objective-driven Adaptive Hybrid Lossy Compression Framework for Extreme-Scale Scientific Applications, G2900453181, G3075337988, G3083819904, G3136028036, G3318903820, Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements, G3886122786, G4501827968, CDS&E: Collaborative Research: HyLoC: Objective-driven Adaptive Hybrid Lossy Compression Framework for Extreme-Scale Scientific Applications, G498139845, G513797252, G5398989114, G5692331544, G6652958823, G6848031779, G6918803902, Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements, G8757742288, G878601127, G969889393, G993149817
OpenAlex ID: W4283748167