Harnessing the Power of HPC From the Comfort of R
Year: 2024
Type: article
Abstract: Many academics feel comfortable wrangling and analyzing data in R, but have little to noexperience working on the command line and may find job scheduling systems like SLURMintimidating. This can be a significant barrier for using high performance computing whichgenerally requires creating BASH scripts and submitting jobs via the command line. The{targets} R package provides many benefits to researchers, one of which is running steps of ananalysis automatically as job requests on an HPC all from the comfort of R.The {targets} package allows for workflow management of analysis pipelines in R wheredependencies among steps are automatically detected1. When a {targets} pipeline is modifiedand re-run, any steps (called “targets”) that do not need to be rerun are automatically skipped,saving compute time. By default, {targets} pipelines are launched in a “clean” R session, whichenforces reproducibility (a blessing to some and a curse to others). It is relatively trivial toparallelize a {targets} workflow so that independent targets are run on parallel workers eitherlocally as multiple R sessions, or using HPC or cloud computing resources2,3. Users can definea controller that runs their pipeline on the HPC using multiple workers running as separateSLURM jobs (or PBS, SGE, etc.). It is also possible to define multiple controllers with differentresources for different targets so that tasks with heavier computational needs are run with moreCPUs, for example. All of this happens from the comfort of R without users needing to manuallycreate multiple R scripts and/or multiple SLURM submission scripts for each task.RSEs and HPC professionals can help enable this powerful combination of technologies in afew ways. At University of Arizona, our group has created a template GitHub repository for a{targets} project that can run on the UA HPC either through the command line where targets arerun as SLURM requests or with Open OnDemand where targets are run on multiple Rsessions4. It includes code for a controller function that works with the UA HPC anddocumentation about how to modify the template and get it onto the HPC with git clone. Wehave previously run {targets} workshops that help researchers re-factor their analysis scripts into{targets} pipelines5. We hope to work with HPC professionals to increase awareness of {targets}as an option for R users to harness the power of cluster computing.
Author: Scott, Eric
Institution: University of Arizona
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6949349101
FWCI:
Cites:
Cited by:
Related to: