An Architecture for Model Monitoring System with Automated Data Validation and Failure Handling
Year: 2025
Type: conference-paper
Abstract: This research paper presents a case study on developing and implementing a comprehensive Model Monitoring system within the banking domain to enhance data quality and establish a robust platform for monitoring cross-domain data. The project, deployed entirely on AWS, focuses on achieving high data accuracy, reliability, and integrity levels in large-scale environments. Leveraging cloud-native services such as AWS EMR, Lambda, S3, SNS, and SQS, alongside tools like Python, PySpark, and Databricks, the system integrates automated data validation, real-time alerting, and scalable monitoring functionalities. The proposed solution combines dynamic and static validation checks, fault handling mechanisms, and proactive notifications to ensure seamless data pipeline operations. This study demonstrates how cloud-based architectures can effectively address data quality challenges and streamline model monitoring in complex and distributed data ecosystems.
Institution: Galgotias College of Engineering & Technology
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412404782