AI-Driven Data Observability: A Hybrid Approach Using Graph Neural Networks and Bayesian Anomaly Detection
Year: 2024
Type: article
Abstract: In modern data-intensive enterprises, ensuring reliability, accuracy, and trust in data pipelines is critical for downstream analytics and decision-making. Traditional monitoring systems often struggle to detect complex dependencies and subtle anomalies across distributed data ecosystems, leading to delayed remediation and increased operational risk. This research introduces a hybrid AI-driven data observability framework that combines Graph Neural Networks (GNNs) for structural dependency modeling with Bayesian anomaly detection for probabilistic uncertainty estimation. The proposed approach leverages GNNs to capture high-dimensional relational patterns across datasets, workflows, and metadata, while Bayesian inference provides robust anomaly detection under uncertainty and sparse conditions. Experiments conducted on enterprise-scale synthetic and real-world datasets demonstrate significant improvements in anomaly detection accuracy, early incident identification, and false-positive reduction compared to conventional rule-based and statistical monitoring techniques. By unifying structural learning with probabilistic reasoning, this hybrid framework enhances trust in data pipelines, reduces mean-time-to-detection (MTTD), and ensures resilient data operations. The findings highlight the potential of AI-driven observability as a foundational capability for next-generation data reliability engineering.
Authors: Ranjeet Kumar, Mohan Vamsi Musunuru, Swaminathan Sethuraman
Institutions: Amazon (United States), Visa (United States)
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7081996537
FWCI:
Cites:
Cited by:
Related to: