Year: 2022
Type: conference-paper
Abstract: With the increasing prevalence of scalable file systems in the context of High Performance Computing (HPC), the importance of accurate anomaly detection on runtime logs is increasing. But as it currently stands, many state-of-the-art methods for log-based anomaly detection, such as DeepLog, have encountered numerous challenges when applied to logs from many parallel file systems (PF-Ses), often due to their irregularity and ambiguity in time-based log sequences. To circumvent these problems, this study proposes ClusterLog, a log pre-processing method that clusters the temporal sequence of log keys based on their semantic similarity. By grouping semantically and sentimentally similar logs, this approach aims to represent log sequences with the smallest amount of unique log keys, intending to improve the ability for a downstream sequence-based model to effectively learn the log patterns. The preliminary results of ClusterLog indicate not only its effectiveness in reducing the granularity of log sequences without the loss of important sequence information but also its generalizability to different file systems’ logs.
Authors: Chris Egersdoerfer, Di Zhang, Dong Dai
Institution: University of North Carolina at Charlotte
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: SHF: Small: A Hybrid NVM based Computing Architecture for Machine Learning Applications, CNS Core: Small: Moving Machine Learning into the Next-Generation Cloud Flexibly, Agilely and Efficiently, SHF: Small: Collaborative Research: A Parallel Graph-Based Paradigm for HPC Parallel File System Checkers
OpenAlex ID: W4320060562
