Improving Anomaly Detection in the HDFS Dataset with Novel Machine Learning Models and Techniques
Year: 2026
Type: article
Abstract: With the growing scale and complexity of log data, manual anomaly detection has become increasingly time-consuming and error-prone, necessitating the development of robust machine learning-based solutions. The HDFS (Hadoop Distributed File System) dataset, a large-scale real-world log collection, serves as a standard benchmark for evaluating both supervised and unsupervised anomaly detection methods. Available in two variants - a reduced and a complete version - this dataset facilitates comprehensive performance comparisons. Our empirical analysis reveals significant limitations in existing approaches: supervised methods exhibit poor performance on the reduced dataset, while most unsupervised techniques underperform across both versions. To address these shortcomings, we introduce several novel machine learning approaches for log-based anomaly detection. Additionally, we investigate the effects of alternative feature extraction techniques. We also examine the application of Synthetic Minority Over-sampling Technique (SMOTE) to mitigate class imbalance in supervised learning, as well as the incorporation of temporal features encoding inter-log time intervals. Our experimental results demonstrate that the proposed methods achieve statistically significant improvements in detection accuracy over existing approaches on both HDFS dataset variants, establishing new benchmarks for log-based anomaly detection.
DOI: 10.56415/csjm.v33.16
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7119022538
FWCI:
Cites:
Cited by:
Related to: