Work

Bulid a Data-pipeline to Process & Analyze Data

Year: 2025

Type: article

Abstract: This project focuses on building a robust data pipeline for processing movie and TV show data stored as CSV files on GitHub. The data is first ingested into Azure Data Lake Storage, forming the base for further processing. Using Databricks Auto Loader, the pipeline supports incremental data loading, allowing it to handle new data efficiently. The pipeline adopts a multi-layered architecture: Bronze Layer: Stores raw data. Silver Layer: Contains cleaned and transformed data. Gold Layer: Hosts refined datasets ready for reporting and analytics. Delta Live Tables (DLT) are used in the Gold layer to ensure streamlined data management and maintain data quality throughout the pipeline.

Source: INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT

DOI: 10.55041/ijsrem50793

Authors: Fabtach Collage Of Engineering Sangola, Rohan Shivaji Bhosale

Language: English

Topic: Data Quality and Management

Subfield: Management Science and Operations Research

Field: Decision Sciences

Domain: Social Sciences

Open Access status: hybrid

OpenAlex ID: W4411418853

FWCI:

Cites:

Cited by:

Related to: 10