Work

Building a Scalable ETL Pipeline with Apache Spark, Airflow, and Snowflake

Year: 2025

Type: article

Abstract: Extract, Transform, and Load (ETL) pipelines are critical in modern data engineering, enabling efficient data integration and analytics. This paper presents a scalable ETL pipeline leveraging Apache Spark for distributed data processing, Apache Airflow for workflow orchestration, and Snowflake as a cloud-based data warehouse. The proposed architecture ensures fault tolerance, cost efficiency, and high scalability, making it suitable for handling large-scale enterprise data workloads.

Source: International journal of innovative research and creative technology

DOI: 10.5281/zenodo.15125061

Author: Ujjawal Nayak

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

SDG: Industry, innovation and infrastructure

Open Access status: hybrid

OpenAlex ID: W6912128997

FWCI: 2.156

Cites:

Cited by: 2

Related to: