Work

Towards Efficient Job Scheduling for Cumulative Data Processing in Multi-Cloud Environments

Year: 2025

Type: article

Abstract: The rapid expansion of multi-cloud environments enables the fulfillment of the dynamic and diverse resource requirements of cloud applications. Cumulative data processing (CDP) applications, which handle incrementally generated data in stages like preprocessing and aggregate analysis, particularly benefit from these environments. However, existing cloud scheduling solutions struggle to handle the dynamic accumulation of processed data and the long-term data operation dependencies in CDP applications. Aiming at this issue, we propose a novel job execution model, CDP-EM, and a tailored job scheduling strategy, CDP-JS, to optimize the scheduling of CDP applications in multi-cloud environments. The CDP-EM model enables dynamic job generation and dependency-aware execution for CDP applications, while the CDP-JS strategy formulates the job scheduling problem as a Markov Decision Process (MDP), utilizing deep reinforcement learning with Proximal Policy Optimization (PPO) to optimize scheduling decisions. The simulation results show that integrating CDP-EM and CDP-JS reduces the SLA violation rate and resource cost of CDP applications by an average of 34.8% and 23.4%, respectively. Real-world evaluations show average reductions of 27.2% and 31.3%, respectively.

Source: Electronics

DOI: 10.3390/electronics14071332

Authors: Yi Liang, Guimei Xu, Haotian Shen, Nianyi Ruan, Yinzhou Wang

Institution: Beijing University of Technology

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,896

Funder: National Natural Science Foundation of China