Work

Data Observability Framework for Mission-Critical Industrial ETL Pipelines: Anomaly Detection, SLA Modeling, and Proactive Failure Identification

Year: 2026
Type: article
Abstract: Traditional monitoring tools can confirm that an ETL pipeline has executed successfully, but they often fail to detect whether the data itself is accurate and complete. In industrial environments, pipelines may finish without errors while delivering stale, incomplete, or incorrect data, creating significant operational risks. To address this challenge, this paper presents the Industrial Observability Platform Framework (IOPF), a five-dimensional observability model comprising freshness modeling, volume anomaly detection, SLA proximity alerting, failure signature classification, and root-cause diagnostic instrumentation. The framework was implemented in a utility-sector production environment processing over 500 million rows daily across 20+ source systems and 2,000+ Oracle Data Integrator (ODI) mappings. During a 180-day evaluation period, IOPF reduced Mean Time to Resolution (MTTR) from 3–4 hours to 15–30 minutes, decreased business-reported incidents from 20–25 to 3–6 per month, and increased proactive incident detection from 10–15% to 65–80%. IOPF extends previous OT/IT integration and cloud-scale ELT frameworks by providing the observability capabilities required for reliable industrial data pipeline operations at enterprise scale.
Author: Ekta Sojitra
Language: English
Open Access status: bronze
OpenAlex ID: W7163325467
FWCI:
Cites: 12
Cited by:
Related to: 10