Predicting computationally critical islands for survey-scale radio source finding with PyBDSF
Year: 2026
Type: article
Abstract: Next-generation radio interferometers require increasingly efficient processing of large image datasets. In highly parallel source-finding workflows, a few computationally expensive PyBDSF islands can dominate the fitting critical path and limit end-to-end throughput. We investigate whether such islands can be identified before fitting and whether this information can improve survey-scale throughput without changing the scientific workflow. We predicted a 90th-percentile (Q90) fitting-time priority from pre-fitting island properties using a histogram-based gradient-boosting quantile-regression model (HGBR-Q90) and supplied the learned priorities to a longest-processing-time (LPT)-based dispatch mechanism. We refer to this approach as computational long-tail prioritisation (CLP). We evaluated CLP with 32 workers on 1580 LoTSS DR3 images, compared it with controlled PyBDSF v1.13.0 baselines, integrated it into PyBDSF v1.14.1 as v14-CLP, and tested generalisation on five external datasets without retraining. The slowest 1% of islands contribute a median of 72.2% of the island-fitting runtime. In the controlled v1.13.0 benchmark, CLP reduces accumulated wall time from 265.85 to 169.23 h and saves 22.99 h relative to deterministic Weight-LPT. In PyBDSF v1.14.1, v14-CLP reduces accumulated wall time from 190.84 to 117.97 h, or 38.2%, while the median and 95th-percentile image runtimes decrease from 314.98 to 202.67 s and from 1015.22 to 635.85 s. Without retraining, B4 is closer to the measured-runtime reference than B3 on all five external datasets and is fastest within the controlled B1--B4 framework on four; v14-CLP further reduces complete-workflow wall time relative to B4 on all five. The computational long tail of PyBDSF island fitting is sufficiently predictable from pre-fitting properties to improve task prioritisation at survey scale. The largest gains occur for task-rich, long-tailed workloads in which island fitting contributes materially to total wall time. Catalogue outputs remain identical because CLP changes only task prioritisation and worker assignment.
Source: Astronomy and Astrophysics
Language: English
Subfield: Astronomy and Astrophysics
Field: Physics and Astronomy
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,865
OpenAlex ID: W7224090562