Hands-on Tutorials
My first-ever publication on Medium was a deep dive into convolutional neural networks (CNN). In that post, I went through a step-by-step example on how to use this technique for medical image classification. It’s a powerful tool in the arsenal of any dataphile, and it’s important to recognize the CNN is not limited to computer vision tasks. Today, I will show you how this technique can be adapted for 1 dimensional sequential data.
It shouldn’t be too shocking that CNNs can be adapted for 1D data. After all, an image is also a sequence of data. The only difference is the 2 dimensional structure (or 3 for color images). This flexibility makes it suitable in a variety of applications, some of which are outlined here.
The goal for today’s post is to give you the tools to implement this technique on any sequential dataset that may interest you. We will look at how to set up our network as a sequential and multi-headed model and compare the results. If you wish to follow along with my notebook, you can find it here. Now, without further ado, let’s get to it!
Libraries
Below, you will find a list of the libraries I used for today’s analyses. They consist of the standard data science toolkit, a scaler from sklearn, and the necessary keras/tensorflow libraries.
Data
Today’s dataset is Kaggle’s Mobile health human behavior analysis which has a CC0: Public Domain license. It consists of Shimmer2 wearable sensors on the right wrist and left ankle of 10 participants. Each sensor collects acceleration and rotational velocity in the x, y, and z planes, resulting in 12 predictor variables.
These variables will be used to classify the participants’ behavior as one of the 12 following activities:
- Standing still (1 minute)
- Sitting and relaxing (1 minute)
- Lying down (1 minute)
- Walking (1 minute)
- Climbing stairs (1 minute)
- Forward folds (20x)
- Frontal elevation of arms (20x)
- Crouching (20x)
- Cycling (1 minute)
- Jogging (1 minute)
- Running (1 minute)
- Jumping forwards and backwards (20x)
The original dataset also contained data from an electrocardiogram and magnetometer, but weren’t available on the Kaggle version. This led me to the decision to remove sitting and lying down activities. The reason for this is due to the lack of magnetometer data, without which standing, sitting, and lying down would have no acceleration and no rotational velocity making them indistinguishable.
The result is the following dataset:
Let’s explore our data a little to see what we find. Using df.info() __ gives us a quick overview of our data frame. Here we see that we have a very tidy data set with no missing values.
Next, we will take a peek at the classification variable using df.Activity.value_counts(). Interestingly, we have the introduction of a new class 0, which is not mentioned in the Kaggle data description.
Because I have no idea what this class refers to, I decided to banish it to digital purgatory using df = df[df.Activity != 0].
One final step to help our model learn is to create even group sizes using the code below. This will reduce any chance of over/under representation influencing our model. It also won’t harm the model because we aren’t removing valuable information. After all, running should look the same whether its 20 seconds or 1 hour (provided the runner doesn’t fall).
Splitting, scaling, and shaping
The last thing we need to do is prepare the dataset so it’s in the appropriate format for our neural network. The code below can be used to split and scale the data. Let’s unpack what it says.
The first two blocks are splitting our data set into 70% for training and 30% for testing. I also further split the data into X (predictors/features) and y (outcome) variables. You could also achieve the same result using sklearn’s train_test_split.
In the last two blocks, we are scaling the data. For the predictors, sklearn’s MinMaxScaler was used to transform our data to fit in the range 0 to 1. It does this by subtracting each value by the minimum and then dividing that by the range (x -min)/(max -min).
The outcome variable was one-hot encoded using Keras’ to_categorical function. This converts our classes into a 1 * N binary matrix (where N is the number of classes). This is an important step when there is no ordered nature to the classes. If your data has an ordered structure (i.e., temperature, age, mass) you could consider using an ordinal encoder instead of a one-hot encoder.
Our CNN has certain requirements regarding the shape of the input data. Below, you will see how we can reshape our data into 3d chunks of input, such that the predictors and output are organized into windows. The window size of 50 was selected because of the 50Hz sampling rate of the accelerometers and gyroscopes used to collect the data described on Kaggle.
With our data in the appropriate format, we can now build our classification model!
CNN
Sequential
Building a neural network is a highly iterative process which requires fine-tuning of multiple hyperparameters to optimize the results. It also includes trying out various architectures. Today we are going to start by building a sequential CNN. It will consist of 2 convolution layers, 1 dropout layer, 1 max pooling layer, 1 flatten layer, 1 dense connected layer, and our classification layer (fig 1).
You could build a hard-coded model and adjust it after each trial, or you could build a function to run multiple trials sequentially as in the code below. I learned this from Jason Brownlee and it’s a great time-saver.
In the code below, we are defining a function _fit_evaluatemodel that will (you guessed it!) fit and evaluate our model. Notice on line 2, the input expects our X and y train/test sets in addition to _nfilters. Using the architecture described above, we will evaluate the following filter maps [8, 16, 32, 64, 128, 256]. We will be tuning the model as we go, so the best performing filter map will be hard-coded into the next round when we tune the kernel size and the dropout rate, as you will see.
Each trial will be running for 10 iterations, each with a batch size of 32. Using a separate function, _performancesummary, we will collect the results and plot them. Each filter map will be run 5 times and we will then calculate the mean and standard deviation of these trials to determine our winner (lines 4–6 below).
Finally, a function _runtrials will tie everything from above together and run the trials we have requested.
The results from this set of trials can be seen below. There is a tendency for the performance to increase with an increase in filter maps. This increase appears to plateau around 32 filter maps. The sweet spot seems to be 64 filter maps, which resulted in the top performance and also the lowest variance meaning it’s the most stable.
Param=8: 60.379% (+/-3.593)
Param=16: 65.084% (+/-2.929)
Param=32: 71.039% (+/-5.879)
Param=64: 73.601% (+/-2.396)
Param=128: 72.013% (+/-5.594)
Param=256: 70.702% (+/-5.059)With an idea of the optimal filter maps, we will next look at tuning the kernel size. If you are unclear on these hyperparameters, don’t forget to check out my article that I linked at the beginning. There are many other good resources, but I went deep into architecture, optimizers, activation functions, and hyperparameters, with examples of each.
Most of the code stays identical as evident below, however, you can see a new variable in line 2, _nkernel, which represents the kernel sizes that we will evaluate here [2, 3, 5, 7, 11].
The results from these trials are presented below. It appears that the smaller kernel sizes are picking up useful information to discriminate our classes that our larger kernel sizes are missing. Although the values don’t vary drastically, using a kernel size of 3 give us the best result. With that said, it doesn’t improve our accuracy from our first set of trials. Let’s see if we can do better by changing our dropout!
Kernel=2: 73.285% (+/-4.676)
Kernel=3: 73.585% (+/-4.133)
Kernel=5: 69.066% (+/-1.468)
Kernel=7: 62.953% (+/-3.858)
Kernel=11: 65.167% (+/-2.749)In the code below, which will be our final set of trials for the sequential model, we are testing dropout rates [0.1, 0.3, 0.5, 0.7, 0.9]. To reiterate, here we have hard-coded the tope performing filter maps (64) and kernel size (3).
The results from these trials are presented below. Although we have increased the overall performance by 2%, it’s not due to anything we have manipulated. For each of the previous trials, a dropout of 0.5 was selected at random. The observed improvement is due to the stochastic nature of these algorithms. Often, you will find people addinga parameter, _randomstate, to their models to ensure they get a result that can be replicated. I opted not to include that because it allows us to get an idea of how stable our models really are.
Dropout=0.1: 74.934% (+/-2.552)
Dropout=0.3: 74.862% (+/-5.299)
Dropout=0.5: 75.111% (+/-5.832)
Dropout=0.7: 72.981% (+/-4.504)
Dropout=0.9: 72.297% (+/-2.313)Multihead CNN
As an alternative to the sequential model presented above, this section will offer a different method of using CNNs. Using a multiheaded CNN allows you to add multiple different parameters in parallel. The information learned from each input is then merged (concatenated) prior to being fed into the final dense layer and subsequently the classification layer.
To help visualize this process, refer to figure 2. Three separate heads are used as input, each of which consists of different filter maps, kernel sizes, and dropouts. After each input is flattened, they are concatenated and fed into a dense layer. Finally, the merged results are fed into the classification layer. Let’s see what this does for our accuracy!
As you can see below, the code doesn’t change that much. The difference is how the various inputs are defined, each with different parameters. The first input has 256 filter maps, a kernel size of 2, and a dropout rate of 0.5. The second input has 128 filter maps, a kernel size of 3, and a dropout rate of 0.3. Finally, the last input has 64 filter maps, a kernel size of 5, and a dropout rate of 0.1.
As seen in the figure above, each output is then flattened (line 25) and then concatenated. The rest is similar to the sequential model, where the merged output is fed into a dense layer with 100 neurons and then fed into the classification layer.
The results from the multiheaded approach are presented below. Using the average of five trials, we see that our performance is slightly less (72%) than what we observed using the sequential model (75%).
>#1: 71.229
>#2: 71.903
>#3: 72.020
>#4: 69.389
>#5: 74.481
Accuracy: 71.804% (+/-1.636)Summary
In today’s post we used 1 dimensional CNNs to classify human activity as measured from biosensors. We walked through examples using a sequential and multiheaded approach. We also looked at manipulating the filter maps, kernel size and dropout rate to see if we could improve the model accuracy.
We finished with an accuracy of 75%, which isn’t too bad, but it’s nothing to write home about. Of course, there are ways to improve our results. In the next post, I will show you how to combine the strengths of CNNs with another specialist in sequence/time series data.
I hope you enjoyed today’s post! Until the next time, Thanks for reading!
