Anomaly Detection in mobile sensor data using Machine Learning

This post is an excerpt from our solution tutorial — “Gather, visualize, and analyze IoT data”. The tutorial walks you through setting up an IoT device, gathering mobile sensor data in the Watson IoT Platform, exploring data and creating visualizations and then using advanced machine learning services to analyze data and detect anomalies in the historical data.

So, what is Anomaly Detection?

Anomaly detection is a technique used to identify unusual patterns that do not conform to expected behavior, called outliers. It has many applications in business, from intrusion detection (identifying strange patterns in network traffic that could signal a hack) to system health monitoring (spotting a malignant tumor in an MRI scan), and from fraud detection in credit card transactions to fault detection in operating environments.

In our day-to-day life, knowingly or unknowingly, We carry an IoT device. It is our mobile phone with inbuilt sensors which provides data from accelerometer and gyroscope. How about saving this sensor data somewhere and detect anomalies in that data?

That sounds like a cool idea. How can we achieve this? Do I need to code an app and ask users to download it from the store? Not required. A simple node.js application running on a mobile browser will provide us with the sensor data.

This tutorial uses the following IBM Cloud products:

• IBM Watson™ IoT Platform
• Node.js Application
• Watson™ Studio
• IBM Analytics Engine
• IBM Cloud Object Storage
• IBM Cloudant

Here’s the flow or architecture diagram,

Architecture diagram

So, you will create a node.js application, run that on a browser, store the accelerometer and gyroscope data to Cloudant NoSQL DB and then how do I detect Anomalies?

Here’s where IBM Watson Studio comes handy. You will use the Jupyter Notebook that is available in the IBM Watson Studio service to load your historical data and detect anomalies using z-score. You will start by creating a new project and then import the Jupyter notebook(.ipynb) through URL.

Anomaly detection will be performed using z-score. Z-score is a standard score that indicates how many standard deviations an element is from the mean. A z-score can be calculated from the following formula: z = (X - µ) / σ where z is the z-score, X is the value of the element, µ is the population mean, and σ is the standard deviation.

The tutorial is kept up-to-date at this link https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-gather-visualize-analyze-iot-data