Year: 2026
Type: article
Abstract: The Smart Data Quality and Auto-Healing Pipeline is an automated, configuration-driven system developed to improve the reliability and usability of structured data, particularly CSV datasets. Data may contain missing values, duplicate records, invalid formats, datatype inconsistencies and unusual observations, making manual cleaning time-consuming and difficult to scale. The pipeline combines ingestion, schema validation, profiling, rule-based validation, optional anomaly detection, safe automatic healing, quality scoring, audit metadata and reporting. YAML configuration separates dataset-specific rules from implementation code. The system validates data before and after healing and records unresolved issues rather than claiming every problem has been fixed. FastAPI exposes pipeline services and cleaned-data downloads, while a Streamlit dashboard presents quality metrics and healing outcomes. In the evaluated sample, the overall quality score increased from 77.02 to 94.13 with 61 automated tests passing.
DOI: 10.64388/ire1724942
Language: English
Topic: Data Quality and Management
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7225571057