Designing and Scaling Real-Time Data Pipelines with Azure Data Factory and Machine Learning Models
Year: 2025
Type: article
Abstract: Azure Data Factory (ADF) provides an exceptional approach to constructing highly elastic, real-time data processing pipelines for machine learning models. This paper offers an architectural approach and recommendations on ADF in the implementation of a cost-effective, high-performance, and real-time data pipeline. The main goal is to show how ADF can be adopted to implement event-based data transfer effectively and to integrate and train machine learning models on real-time feeds. This paper gives a pipeline evaluation where the pipeline has a throughput of 95,000 events/sec with a test latency of 500 milliseconds at optimum loads thus indicating better efficiency compared to similar systems. According to these research findings, ADF-based pipelines provide the benefit of utilizing up to 30% less resources than the classic batch-processing architectures. Furthermore, this study offers a conceptual model for processing real-time information that boosts decision-making abilities in manufacturing firms, the healthcare sector, and the retail industry.
Author: Siva Karthik Devineni
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6911900289
FWCI:
Cites:
Cited by:
Related to: