Skip to content
/ MTAD Public

MTAD: Tools and Benchmark for Multivariate Time Series Anomaly Detection

Notifications You must be signed in to change notification settings

OpsPAI/MTAD

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MTAD: Tools and Benchmark for Multivariate Time Series Anomaly Detection

This repository is a Multivariate Time Series Anomaly Detection toolkit named MTAD with a comprehensive benchmarking protocol and contains state-of-the-art methods with a unified and easy-to-use interface. We include 15 methods in our repo, which are evaluated on 4 public datasets.

Citation

👋 If you use our tools or benchmarking results in your publication, please cite the following paper.

Jinyang Liu, Wenwei Gu, Zhuangbin Chen, Yichen Li, Yuxin Su, Michael R. Lyu. MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection.

Our evaluation protocol

Our evluation protocal can be summarized as the following figure, where we consider different threshold selection strategies and prediction adjustment. We also include the evaluation of delay in our protocol.

Alt text

Threshold Selection:

  • EVT-based method
  • Seaching for the optimal one

Metrics:

  • Precision, Recall, F1-score: how accruate a model is?
    • with or without point adjustment
  • Delay: how timely can a model report an anomaly?
  • Efficiency: how fast can a model be trained and perform anomaly detection?

Requirement

cd ./requirements

pip install -r <requirement file>

Run our benchmark

The following is an example to run benchmark for LSTM, whose configuration files are stored in the ./benchmark_config/folder.

cd benchmark
python lstm_benchmark.py

Models integrated in this tool

General Machine Learning-based Models

Model Paper reference
PCA [2003] Shyu M L, Chen S C, Sarinnapakorn K, et al. A novel anomaly detection scheme based on principal component classifier
iForest [ICDM'2008] Fei Tony Liu, Kai Ming Ting, Zhi-Hua Zhou: Isolation Forest
LODA [Machine Learning'2016] Tomás Pevný. Loda**:** Lightweight online detector of anomalies

Deep Learning-based Models

Model Paper reference
AE [AAAI'2019] Andrea Borghesi, Andrea Bartolini, Michele Lombardi, Michela Milano, Luca Benini. Anomaly Detection Using Autoencoders in High Performance Computing Systems
LSTM [KDD'2018] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, Tom Söderström. Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
LSTM-VAE [Arxiv'2017] A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder
DAGMM [ICLR'2018] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae-ki Cho, Haifeng Chen. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection
MSCRED [AAAI'19] Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, Nitesh V. Chawla. A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data.
OmniAnomaly [KDD'2019] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, Dan Pei. Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network
MTAD-GAT [ICDM'2020] Multivariate Time-series Anomaly Detection via Graph Attention Networks
USAD [KDD'2020] USAD: UnSupervised Anomaly Detection on Multivariate Time Series.
InterFusion [KDD'2021] Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, Dan Pei. Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding
TranAD [VLDB'2021] TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data
RANSynCoders [KDD'2021] Practical Approach to Asynchronous Multivariate Time Series Anomaly Detection and Localization
AnomalyTransformer [ICLR'2022] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy
GANF [ICLR'2022] Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series

Datasets

The following datasets are kindly released by different institutions or schools. Raw datasets could be downloaded or applied from the link right behind the dataset names. The processed datasets can be found here⬇️ (SMD, SMAP, and MSL).

  • Server Machine Datase (SMD) Download raw datasets⬇️

    Collected from a large Internet company containing a 5-week-long monitoring KPIs of 28 machines. The meaning for each KPI could be found here.

  • Soil Moisture Active Passive satellite (SMAP) and Mars Science Laboratory rovel (MSL) Download raw datasets⬇️

    They are collected from the running spacecraft and contain a set of telemetry anomalies corresponding to actual spacecraft issues involving various subsystems and channel types.

  • Secure Water Treatment (WADI) Apply here*

    WADI is collected from a real-world industrial water treatment plant, which contains 11-day-long multivariate KPIs. Particularly, the system is in a normal state in the first seven days and is under attack in the following four days.

  • Water Distribution (SWAT) Apply here*

    An extended dataset of SWAT. 14-day-long operation KPIs are collected when the system is running normally and 2-day-long KPIs are obtained when the system is in attack scenarios.

* WADI and SWAT datasets were released by iTrust, which should be individually applied. One can request the raw datasets and preprocess them with our preprocessing scripts.

Releases

No releases published

Packages

No packages published

Languages