Optimizing Distributed Machine Learning Workloads and Resource Management in Multi-cloud Environments Under Various Workload Conditions
Year: 2025
Type: preprint
Abstract: Distributed machine learning leverages multiple computing resources to process large datasets efficiently. However, running workloads across multi-cloud environments presents challenges like varying resource availability, high communication latency, and cost inefficiencies. This study addresses the problem of optimizing workload distribution and resource management in heterogeneous multi-cloud setups. The objective is to develop a dynamic framework that improves execution speed, reduces communication overhead, and lowers operational costs, ensuring scalable and efficient distributed machine learning across diverse cloud platforms. This study models a heterogeneous multi-cloud environment using AWS, Google Cloud, and Azure, leveraging containerization and Kubernetes for scalable deployment. Large datasets are efficiently partitioned and distributed to minimize inter-cloud communication and enhance fault tolerance. An Adaptive Scheduling using Reinforcement Learning Optimization (ASRLO) dynamically balances workloads based on real-time patterns. Compressed Routing and Messaging Architecture (CRMA) reduces communication overhead, while Real-time Elastic Monitoring with Predictive Adjustment (REMPA) ensures proactive resource management. Together, these techniques optimize distributed machine learning workloads for improved performance, scalability, and cost-efficiency in multi-cloud settings. Findings show that the optimized multi-cloud framework significantly enhances distributed ML workloads by reducing operational costs from 50 to 30 hours, lowering data transfer from 200 GB/hr to 120 GB/hr, and increasing availability to 99.5%. OBSMO achieves the lowest VM cost (18–79) and energy consumption (1.5–7 KJ). The proposed method offers stable convergence (0.59–0.528) and scalability, implemented in Python Software. Future scope includes refining accuracy, reducing inter-cloud latency, and balancing energy efficiency with performance in real-time, high-variance multi-cloud environments.
Authors: Babasaheb Annasaheb Abhale, Somnath Ashok Gade, Sachin Ramnath Kolase
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4413411517