Work

Adaptive Workflow Orchestration with Reinforcement Learning for Optimized Data Processing

Year: 2026
Type: conference-paper
Abstract: Modern data processing systems rely on complex workflows consisting of multiple interdependent tasks executed across distributed computing environments. Efficient orchestration of these workflows is critical for optimizing resource utilization, reducing latency, and ensuring scalability. Traditional scheduling approaches often rely on static rules or heuristics, which are insufficient for dynamic and heterogeneous workloads. In this paper, we propose an adaptive workflow orchestration framework based on reinforcement learning (RL) to optimize data processing pipelines. The proposed approach models workflow scheduling as a sequential decision-making problem, enabling the system to learn optimal policies for task allocation and execution. The framework dynamically adapts to changing workloads and system conditions, improving efficiency and performance. Experimental results demonstrate that the RL-based approach significantly reduces execution time and resource consumption compared to traditional methods. This work highlights the potential of reinforcement learning for intelligent workflow management in distributed data processing systems.
Language: English
Open Access status: hybrid
OpenAlex ID: W7171531430
FWCI:
Cites: 9
Cited by:
Related to: 10