Year: 2026
Type: article
Abstract: Radio astronomy observations generate massive heterogeneous datasets with file sizes ranging from kilobytes to gigabytes and total volumes reaching terabytes per session. Telescope sites in remote areas face limited network connectivity, making efficient data transfer a significant challenge. The conventional rsync tool, while widely used for incremental transfer, suffers from serial limitations that prevent full bandwidth utilization, particularly for extreme file-size distributions characteristic of radio astronomy data. This paper presents a heterogeneity-aware parallel data transfer method built upon the rsync framework, designed for datasets from facilities such as the Qitai radio telescope (QTT). Our approach introduces three key contributions: (i) a heterogeneity-aware task partitioning strategy with an adaptive correction factor κ ( C v ) that isolates files into singleton chunks; (ii) a mathematical performance model quantifying the impact of data heterogeneity on parallel speedup through a load imbalance factor γ ( N , C v , ρ ) ; and (iii) a checkpoint-based fault tolerance mechanism that eliminates costly directory re-traversal overhead upon interruption. Experimental evaluation on both simulated and authentic ASKAP observation data demonstrates the effectiveness of our method. On the representative data-mixed workload, our method outperforms Optimized Rsync by 16.8% on simulated data and by 28.8% on real data, while maintaining CPU utilization below 1.14% and a memory footprint approximately three times smaller than Standard Rsync. The efficiency ratio shows at least two orders of magnitude higher resource efficiency than all baseline tools. The proposed method offers an alternative transmission solution for the QTT data transfer system, seamlessly integrating with the existing rsync ecosystem while consuming minimal computing resources.
Source: Astronomy and Computing
Authors: Jie Wang(Conceptualization, Methodology, Software, Writing – original draft), Hai-long Zhang(Methodology, Writing – review & editing), Bo Wang(Software, Validation), Shi Su(Investigation, Resources), Xin-chen Ye(Investigation, Resources), Ya-zhou Zhang(Data curation, Visualization), Hong-mei Tang(Formal analysis, Software), Ting Zhang(Resources, Validation), Wan-qiong Wang(Data curation, Investigation), Jia Li(Software, Visualization), Wen-na Cai(Data curation, Writing – original draft), Yu-yue Jiao(Writing – original draft)
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,260
OpenAlex ID: W7234382608
