Year: 2025
Type: article
Abstract: Modern financial enterprises face growing complexity in managing high-volume, high-velocity, and high-variety data generated by various channels, including trading platforms, mobile banking, credit scoring engines, and compliance systems. Traditional Extract-Transform-Load (ETL) mechanisms are increasingly strained under these demands, leading to performance bottlenecks, data latency, and governance risks. This paper presents a comprehensive review and architectural model for optimizing ETL pipelines to support scalable Business Intelligence (BI) in the financial sector. Drawing upon 30 peer-reviewed sources, we analyze challenges such as real-time processing, metadata management, observability, and regulatory compliance. We propose a modern ETL reference architecture using tools such as Apache Airflow, Kafka, dbt, Spark, and cloud-native data warehouses. Benchmark evaluations show performance gains of 60–90% in load time and 95% improvement in pipeline reliability over legacy systems. This study offers an actionable roadmap for financial institutions aiming to modernize their data infrastructure in alignment with evolving regulatory and business intelligence needs.
Author: Laxmi Vanam
Institution: The New World Foundation
Language: English
Subfield: Management Information Systems
Domain: Social Sciences
Open Access status: diamond
OpenAlex ID: W7083590370