Why Benign-Only Log Anomaly Detection Fails Under Drift: Two Mechanisms, Seven Designs, and a Label-Efficiency Bound
Year: 2026
Type: preprint
Abstract: Deep sequence models dominate log-based anomaly detection benchmarks, yet controlled re-evaluations have shown that their reported accuracy collapses under chronological evaluation---without explaining why. We dissect that collapse. Under a leak-free, benign-only protocol on HDFS, BGL, and Thunderbird---chronological and random splits side by side, ten-seed confidence intervals, DeepLog and LogBERT re-run on byte-identical splits---lightweight CPU detectors match or exceed the deep re-runs wherever the template vocabulary is stable (F1 0.92--0.97 at 0.2--0.5\% false-positive rates), and on two of the three benchmarks a one-line unseen-template rule suffices (F1 0.90--0.998), a quantified form of the dataset-triviality critique; HDFS, whose anomalies are structural, is the one system that needs a learned detector. The chronological collapse has two distinct mechanisms. On BGL the template vocabulary is replaced mid-trace: identity evidence is destroyed, the only transferable signal is discriminative fault semantics, and benign-only learning cannot exploit it---six designs without external semantic priors fail concordantly, a frozen pretrained-embedding prior lifts ranking above chance without becoming deployable, and full period-causal supervision yields a ranking (PR-AUC 0.722) whose operating point does not transfer (F1 0.44 causal versus 0.76 oracle). On Thunderbird the signal survives but the calibration era does not: a labeled counterfactual bounds what era-matched recalibration could recover (PCA F1 0.61). Both failures are silent at the alarm level; we identify two label-free self-monitoring statistics that expose them. All code, splits, per-seed results, and re-run artifacts are released.
Source: Research Square
Author: Sarita Nandal
Institution: Indian Institute of Technology Roorkee
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234726062
