A Machine Learning Framework for Automated Data Cleaning and Anomaly Detection in Large Datasets
Year: 2025
Type: article
Abstract: High-quality data is essential for robust machine learning applications, yet large datasets are often compromised by anomalies, missing values, and inconsistencies. This study proposes a novel machine learning framework for automated data cleaning and anomaly detection, integrating dimensionality reduction, anomaly detection, and data imputation techniques. The framework employs Isolation Forests and Autoencoders for anomaly detection, Principal Component Analysis (PCA) and t-SNE for dimensionality reduction, and Random Forest and deep generative models for imputing missing or erroneous data. Evaluated on diverse real-world datasets from finance, healthcare, and manufacturing, the framework achieves high precision (up to 0.88) and F1-scores (up to 0.84) in anomaly detection and low Mean Absolute Error (as low as 0.015) in imputation, significantly enhancing data quality and downstream model performance. The results underscore the framework’s applicability across domains, reducing manual preprocessing efforts. Future research will focus on extending the framework to real-time data streams and exploring domain-specific anomaly correction strategies.
Institution: Chameli Devi Group of Institutions
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7218304743
