Machine Learning Anomaly Detection Service

Azure Machine Learning provides data scientists with a channel to monetize their know-how and creativity just as smartphone app developers do. By publishing your machine learning services in the Azure Marketplace, you enable anyone in the world to pay and subscribe to your published cloud APIs and use them in their apps.

But Azure Marketplace can offer much more than basic analytics APIs. For example, we built a set of finished end-to-end Machine Learning applications to solve specific business needs. These Machine Learning apps do not require a data scientist or Machine Learning expertise to use them. The science is already baked into our solutions. Users can simply supply their own data and start using the apps.

For example,

Frequently Bought Together: An API that helps your customers discover items in your catalog that are frequently purchased together. Use your customer purchase history to add Frequently Bought Together recommendations to your website and to improve conversion in your digital store.

Anomaly Detection: An API that can spot anomalous events in time series data (for example, server telemetry or sensor-based data).

These APIs allow you to consume powerful machine learning-based services built on Azure Machine Learning, without needing to be a data scientist or spend a lot of time developing apps yourself. Simply bring your own data and start using them.

http://azure.microsoft.com/en-in/documentation/articles/machine-learning-apps-anomaly-detection/

anomalies in time series data with numerical values that are uniformly spaced in time.
This anomaly detection service can detect the following different types of anomalies on time series data:
1. Positive and negative trends: When monitoring memory usage in computing, for instance, an upward trend is indicative of a memory leak,
2. Increase in the dynamic range of values: As an example, when monitoring the exceptions thrown by a service, any increases in the dynamic range of values could indicate instability in the health of the service, and
3. Spikes and Dips: For instance, when monitoring the number of login failures to a service or number of checkouts in an e-commerce site, spikes or dips could indicate abnormal behavior.

These detectors track changes in values over time and reports ongoing changes in their values. They do not require adhoc threshold tuning and their scores can be used to control false positive rate. The anomaly detection API is useful in several scenarios like service monitoring by tracking KPIs over time, usage metrics such as number of searches, numbers of clicks, performance counters like memory, cpu, file reads, etc. over time.

The service provides a REST based API over HTTPS that can be consumed in different ways including a web or mobile application, R, Python, Excel, etc. We have an Azure web application that helps run the anomaly detection web service on your data and visualize the results.

You can also send your time series data to this service via a REST API call, and it runs a combination of the three anomaly types described above. The service runs on the AzureML Machine Learning platform which scales to your business needs seamlessly and provides SLAs of 99.95%.

The figure below shows an example of anomalies detected in a times series using the above framework. The time series has 2 distinct level changes, and 3 spikes. The red dots show the time at which the level change is detected, while the red upward arrows show the detected spikes.

The API takes 2 input parameters

1. "data" represents input time series in the format: t1=v1;t2=v2;...

   Sample data:

   "9/21/2014 11:05:00 AM=3;9/21/2014 11:10:00 AM=9.09;9/21/2014 11:15:00 AM=0;"

2. "params" set to "SpikeDetector.TukeyThresh=3; SpikeDetector.ZscoreThresh=3" which controls sensitivity of spike detectors. Higher values will catch higher spikes and vice versa.

   Sample URL with input parameters mentioned above:

   https://api.datamarket.azure.com/data.ashx/aml_labs/anomalydetection/v1/Score?data=%279%2F21%2F2014%2011%3A05%3A00%20AM%3D3%3B9%2F21%2F2014%2011%3A10%3A00%20AM%3D9.09%3B9%2F21%2F2014%2011%3A15%3A00%20AM%3D0%3B%27&params=%27SpikeDetector.TukeyThresh%3D3%3B%20SpikeDetector.ZscoreThresh%3D3%27

Output

The API runs these detectors on your time series data and returns the anomaly scores at each point in time. This output can be parsed using a simple parser as shown in https://adresultparser.codeplex.com/. This gives sample code that shows how to connect to the API and parse the output. The alerts generated can be visualized on a dashboard and/or passed on to human experts who can take corrective actions.

Sample output for the sample input above:

"Time,Data,TSpike,ZSpike,Martingale values,Alert indicator,Martingale values(2),Alert indicator(2),;9/21/2014 11:05:00 AM,3,0,0,-0.687952590518378,0,-0.687952590518378,0,;9/21/2014 11:10:00 AM,9.09,0,0,-1.07030497733224,0,-0.884548154298423,0,;9/21/2014 11:15:00 AM,0,0,0,-1.05186237440962,0,-1.173800281031,0,;"

which is a representation of the following table:

TIME	DATA	TSPIKE	ZSPIKE	MARTINGALE VALUES	ALERT INDICATOR	MARTINGALE VALUES (2)	ALERT INDICATOR (2)
9/21/2014 11:05	3	0	0	-0.687952591	0	-0.687952591	0
9/21/2014 11:10	9.09	0	0	-1.070304977	0	-0.884548154	0
9/21/2014 11:15	0	0	0	-1.051862374	0	-1.1738002814	0

http://anomalydetection-aml.azurewebsites.net/

stream mode

single mode