MetSTAC: Building STAC Catalogs for Machine Learning Ready Data
Year: 2026
Type: article
Abstract: Machine learning workflows depend on access to large and well‑described datasets, yet locating these datasets remains a significant barrier to development. Whilst we have the tools required to build these datasets, they are often forgotten or lost, leading to duplicated effort. Within the Met Office, data used for machine learning spans multiple storage systems and formats, making discovery and reuse difficult The ML Datasets team has adopted the SpatioTemporal Asset Catalog (STAC) standard as a common metadata layer for machine‑learning‑ready meteorological and climate data. This poster presents MetSTAC, a Python library developed to generate STAC catalogs for complex scientific datasets without requiring data duplication or format changes. We show how MetSTAC supports automated metadata extraction and template‑driven catalog generation for both gridded NetCDF data and cloud‑optimised Zarr stores. The poster demonstrates how STAC catalogs provide a single, discoverable view of datasets distributed across local and cloud platforms, enabling users to search and assess data suitability quickly and easily. By aligning with FAIR data principles (Findable, Accessible, Interoperable, and Reusable) MetSTAC illustrates the role of research software engineering in building sustainable metadata infrastructure. While developed in a meteorological context, the approaches demonstrated are applicable to other data‑intensive scientific domains facing similar discovery and reuse challenges.
Authors: Pugsley, Jodeci, Seabourne, Lucy
Institution: Met Office
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204457182
FWCI:
Cites:
Cited by:
Related to: