Work

Publishing Large Datasets: Experiences with Data Scaling and Hybrid Model Publication

Year: 2025

Type: article

Abstract: The publication of large datasets poses significant challenges for research data management, particularly regarding storage, access, citability, and adherence to FAIR principles. We present our experience publishing a 25 TB dataset from the Infrared Atmospheric Sounding Interferometer (IASI) covering October 2014 to June 2019 (MUSICA IASI v3.2.1) using a hybrid approach. A representative one-day subset was deposited in the institutional repository RADAR4KIT with a DOI, while the complete dataset is archived at the Large Scale Data Facility (LSDF) provided by the Scientific Computing Center (SCC) at KIT and accessed via a THREDDS Data Server (TDS) hosted by IMKASF at KIT. We describe the workflow, metadata harmonisation, persistent linking strategy, and lessons learned from this publication process.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.17815952

Authors: Kerzenmacher, Tobias, Barthlott, Sabine, Schneider, Matthias, Bach, Felix, Soltau, Kerstin, Hofmann, Stefan

Institutions: Karlsruhe Institute of Technology, FIZ Karlsruhe – Leibniz Institute for Information Infrastructure

Language: English

Topic: Research Data Management Practices

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Leibniz-Gemeinschaft

Award: G7406238935