Work

pyfive: A pure-Python HDF5 reader

Year: 2026

Type: article

Abstract: pyfive is an open-source and thread-safe pure Python package for reading data stored in HDF5.While it is not a complete implementation of all the specifications and capabilities of HDF5, it includes all the core functionality necessary to read gridded datasets, whether stored contiguously or with chunks (with or without standard compression options).All data access is fully lazy as the data is only read from storage when the numpy data arrays are manipulated.Originally developed some years ago, the package has recently been expanded to support lazy data access, and to add missing features necessary for handling all the HDF5-based environmental data known to the authors.It is now a realistic option for production data access in environmental science and more broadly across other domains.The API is based on that of h5py (https://github.com/h5py/h5py,a Python shimmy over the HDF5 C-library which itself is not thread-safe), with some API extensions to help optimise remote access.With these extensions, coupled with thread safety, many of the limitations precluding the efficient use of HDF5 (and netCDF4) on cloud storage have been removed.

Source: The Journal of Open Source Software

DOI: 10.21105/joss.09688