Year: 2023
Type: article
Abstract: The evolution of Continuous Integration and Continuous Deployment (CI/CD) techniques in contemporarysoftware engineering has significantly revolutionized traditional DevOps techniques. However, existing researchand practices are mostly application-layer deployments without or with insufficient consideration for schemadriven data systems and Kafka-based streaming systems. As a result, there exists a vast research gap in regardsto the provision of intelligent CI/CD pipelines that efficiently manage schema evolution, data validation, andreal-time stream processing for ensuring system integrity and delivery speed. This paper presents an AI-basedCI/CD framework specifically tailored for data-infrastructure schemas and Apache Kafka-based deployments.The framework uses machine learning models to forecast schema drift, identify breaking changes, and performauto-compatibility testing for stream producers and consumers. It is more efficient than conventional rule-basedsystems because the AI models learn to adapt to changing data attributes and deployment patterns, facilitatingproactive anomaly detection and decision-making during CI/CD pipeline execution. Reinforcement learningagents are also present to learn to optimize deployment strategies based on past success/failure patterns. Theapproach also utilizes schema registries, version control, and auto-rollback strategies to support safe andtraceable data schema migration between environments. A simulation test verifies the efficacy of the approachsuggested for minimizing downtime, avoiding data loss, and speeding up deployment cycles for distributed dataplatforms. By solving the long-neglected complexity of data schema lifecycle management and real-time streaminfrastructure, this work fills an important void and opens the door to intelligent CI/CD pipelines specific todata-driven architectures. The results uncover new avenues for scalable, fault-tolerant, and automation-friendlyDevOps practices in the case of data infrastructure engineering.
Author: Raghu Ram Bojanapalli
Institution: University at Buffalo, State University of New York
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7220511715