How to Detect Anomalous Behavior in Time-series

Business and Technology Insights

How to Detect Anomalous Behavior in Time-Series

 
June 4, 2020

We live in a digital world where we cannot imagine our lives without internet, social media, machines, and Internet of Things (IoT) devices. One thing is common across all - data. The IDC white paper predicted that total data creation will reach new heights of nearly 160 ZB in next five years. Data can be in diverse formats and one of the most dominant formats is time-series. Embedded sensors within IoT devices are continuously generating streaming data, which are nothing but sequences of time-series. Availability of such huge amount of time-series data has itself become a challenge because monitoring it is not easy. Let us try to find out the best options available to efficiently monitor and detect unusual behavior in enormous time-series data.

Before diving deep, it is important to understand what anomalous behavior of time-series data is. An anomaly is an outlier in a dataset, which is an observation that deviates significantly from majority observations. For example, in any organization a sudden rise or drop in employee monthly compensation can be an anomalous behavior. Modern machine learning models for anomaly detection in time-series data can be very useful but we cannot deploy a single method in every use-case. Statistical models such as Auto Regressive Integrated Moving Average (ARIMA) are quite effective in some areas whereas deep learning-based anomaly detection techniques have shown promising results in others. When we look for detecting anomaly, the target outcome is a classifier, which aims to distinguish normal and anomalous instances. Models identify anomalous instances of time-series data based on two important features; infrequent and unexpected. Let us discuss some of the best ways to do this.

Statistical Model - ARIMA

ARIMA is a statistical model, which analyzes and estimates subsequent near-future values of a time-series using the dependency of observations and residual errors from a moving average on lagged intervals. ARIMA is suitable for stationary time-series data where mean, variance, autocorrelation are almost constant over a certain period. For example, we could use ARIMA for detecting anomalies in the monthly employee headcount of organization by tracking any unusual gap between actual and forecasted outcome.

Autoencoder

We can utilize Autoencoder to detect outlier from a huge set of data, which is a special unsupervised Artificial Neural Network (ANN). It takes input, condenses the input to core features then reverses the process and recreate the input. In this process, if there is any significant deviation between output and input values for an observation, we can consider it an outlier. An example of real-life autoencoder is fraud detection. Detecting fraud in invoices from supplier involves huge datasets, which requires model to be capable of running multilayer inputs to analyze and detect any anomaly in run-time. In cases where detecting occurrences such as unexpected spikes in pricing, sudden increases in invoice amounts, or several invoices with duplicate invoice number is required, autoencoder is one of the best choices.

LSTM

Long Short-Term Memory (LSTM) is a Recurrent Neural Network (RNN), which processes sequential unstructured data such as video, sound, text or time-series. It is capable of learning long-term dependencies of input data. It consists of a cell, an input gate, an output gate and a forget gate. The three gates control the flow of information whereas cell remembers some values. LSTM is suited to classify and detect anomalies where time-series data is unstructured. LSTM is useful in detecting anomalies in cases such as network traffic or web traffic.

Web traffic is sent and received by a user while visiting any website. Anomalies in web traffic refer to the abnormal values of data with respect to surrounded data. Anomalies in web traffic services may lead to social and economic losses. Web traffic, clickstreams consist of huge datasets of one-dimensional time-series, and LSTM is good for holding long-term memories. Hence, LSTM is effective for network optimization and for web or mobile application fine-tuning using navigation analytics.

Transfer Learning

Transfer learning is cost effective, quick approach to detect anomalous transactions in time-series using deep learning. Using this, we can reuse a model developed for one use-case without building the deep, multilayered network from scratch. The ANN architecture for transfer learning approach for this kind of problem can be one-dimensional Convolution Neural Network. For example, data from accelerometer sensors from smart phones or fitness devices could be analyzed for any anomalies in health conditions of patients or employees. Such networks can also detect anomalies of GPS location sequences of logistics fleets.

Conclusion

Now a day’s time-series anomaly detection is attracting significant interest in industries. Companies use anomaly detection to monitor production process. In this context, methods, which can automatically detect anomalous behavior in the collected data, can have a great impact.

Swayanta Bhaumik is a developer with 3 years of experience in the Innovation and Product Engineering (IPE) - Analytics group in TCS Platform Solutions. He has a good experience of working on Python, Data Science and Machine Learning. He has been contributing to the development and implementation of predictive analytics and embedded analytics use cases. He holds a bachelor’s degree in Electrical Engineering from Maulana Abul Kalam Azad University of Technology (Formerly known as West Bengal University of Technology).