Year: 2017
Type: preprint
Abstract: Scientists are increasingly turning to datacenter-scale computers to produce and analyze massive arrays. Despite decades of database research that extols the virtues of declarative query processing, scientists still write, debug and parallelize imperative HPC kernels even for the most mundane queries. This impedance mismatch has been partly attributed to the cumbersome data loading process; in response, the database community has proposed in situ mechanisms to access data in scientific file formats. Scientists, however, desire more than a passive access method that reads arrays from files. This paper describes ArrayBridge, a bi-directional array view mechanism for scientific file formats, that aims to make declarative array manipulations interoperable with imperative file-centric analyses. Our prototype implementation of ArrayBridge uses HDF5 as the underlying array storage library and seamlessly integrates into the SciDB open-source array database system. In addition to fast querying over external array objects, ArrayBridge produces arrays in the HDF5 file format just as easily as it can read from it. ArrayBridge also supports time travel queries from imperative kernels through the unmodified HDF5 API, and automatically deduplicates between array versions for space efficiency. Our extensive performance evaluation in NERSC, a large-scale scientific computing facility, shows that ArrayBridge exhibits statistically indistinguishable performance and I/O scalability to the native SciDB storage engine.
Source: arXiv (Cornell University)
Authors: Xing, Haoyuan, Floratos, Sofoklis, Blanas, Spyros, Byna, Suren, Prabhat, Wu, Kesheng, Brown, Paul
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: II-New: Infrastructure for Energy-Aware High Performance Computing (HPC) and Data Analytics on Heterogeneous Systems, G122673668, G1286236842, G1677143136, G1799333409, G1863003419, III: Small: A User-centric Architecture for Ad-hoc Analytical Processing, G3136028036, CRII: III: Declarative array processing for large-scale scientific analyses, G419881185, G42916606, G4501827968, G498139845, G5076365615, G6652958823
OpenAlex ID: W2593482382