PyDAP Revisited: Exploiting OPeNDAP's Data-Proximate Transformation Tools to Accelerate Scientific Workflows
Year: 2024
Type: article
Abstract: OPeNDAP’s Hyrax Server enables researchers to access and share scientific datasets over the web efficiently, especially for NetCDF and HDF formats. Despite its widespread adoption and advancements like DAP4 and DMR++, the lack of modern client API tools hampers its usability in contemporary scientific workflows. PyDAP, a Python-based implementation of the DAP protocol, addresses this gap by integrating seamlessly with Xarray, allowing Python-native access to OPeNDAP data. This presentation revisits PyDAP’s modernization, highlighting the challenges and solutions in leveraging DAP4’s capabilities. Recent and ongoing developments include improved hierarchical data handling, enhanced DMR++ parsing for serverless S3 access, and the adoption of FastAPI to boost PyDAP’s performance. Benchmarks demonstrate significant performance gains using constrained requests, achieving up to 10-100x speedup in remote dataset access. These advancements position PyDAP as a critical tool for enabling efficient, scalable, and cost-effective open science workflows.
Authors: Jimenez-Urias, Miguel, Gallagher, James
Institution: OPeNDAP
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6911842886
FWCI:
Cites:
Cited by:
Related to: