HAD-CDA: a hybrid approach for anomaly detection in streaming data with concept drift adaptation

Abstract

The streaming anomaly detection is a difficult task since the data distribution is changing, and concept drift has a negative effect on the performance of the traditional detection methods. To solve the issue, this article suggests the HAD-CDA (Hybrid Anomaly Detection with Concept Drift Adaptation) which is a combined system that can detect local anomalies in the time series with the LSTM Autoencoders (LSTM-AE) and global anomalies in the distribution with Quant Tree-EWMA (QT-EWMA). The proposed framework accomplishes three objectives: (i) the use of the dynamic weighting mechanism, which automatically changes the contribution of each component (lambda between 0.2 and 0.8) according to their effectiveness, (ii) two concept drift detectors are proposed, i.e. the KolmogorovSmirnoff and PageHinkley tests to allow detecting concept drift, and (iii) the use of Elastic Weight Consolidation (EWC) to reduce catastrophic forgetting during update of the model. Experiments on four real world streaming datasets HTTP, SMTP, ForestCover and Shuttle indicate that HAD-CDA has AUCs of 0.95-0.97, a 8-9% improvement in state of the art methods and F1-scores of 0.81-0.95. The recall measure obtained by the LSTM-AE element is 0.8094 as compared to 0.9399, and the specificity of QT-EWMA is very high, 0.9399. The framework is highly adaptable to different types of drifts, regaining around 9092 performance levels before drift, and 1525 windows would be needed by a baseline method, with low processing latency of 12.430.9 ms per window. Having a per-sequence complexity of O(1), memory cost of O(n), low DIS (approximately 0.08), and stability indices of 0.02-0.04, the suggested HAD-CDA framework is an accurate, efficient, and robust solution to real-world streaming anomaly detection in changing data.