Quantum Flow: Enterprise Data Orchestration and Processing Suite
Year: 2024
Type: conference-paper
Abstract: In the realm of data-intensive enterprises, the QuantumFlow project offers a tailored solution for high-performance computing environments, integrating Apache Airflow, Apache Spark, and Apache NiFi. This paper synthesizes extensive research and development efforts to create a robust suite for orchestrating enterprise data workflows efficiently. Emphasizing fault tolerance and scalability, QuantumFlow ensures reliable handling of large-scale data workloads through distributed computing paradigms. Drawing from scholarly works like ExplSched and EVOLVE, informs design decisions and architectural considerations. It delves into practical implementations, challenges, and lessons learned, highlighting the potential to enhance enterprise data engineering capabilities in HPC environments.
Authors: Parish Bindal, Kapil Madan
Institution: Jaypee Institute of Information Technology
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4403882759