How to Implement Deep Neural Networks for Radar Image Classification
Leverage network architectures and techniques developed for computer vision
Introduction
Radar-based recognition and localization of people and things in the home environment has certain advantages over computer vision, including increased user privacy, low power consumption, zero-light operation and more sensor flexible placement.
Shallow machine learning techniques such as Support Vector Machines and Logistic Regression can be used to classify images from radar, and in my previous work, Teaching Radar to Understand the Home and Using Stochastic Gradient Descent to Train Linear Classifiers I shared how to apply some of these methods.
In this article, you will learn how to develop Deep Neural Networks (DNN)and train them to classify objects in radar images. In addition, you will learn how to use a Semi-Supervised Generative Adversarial Network (SGAN) [1] that only needs a small number of labeled data to train a DNN classifier. This is important in dealing with radar data sets because of the dearth of large training sets, in contrast to those available for camera-based images (e.g., ImageNet) which has helped to make computer vision ubiquitous.
Both DNNs (or more specifically Convolutional Neural Networks) and SGANs that were originally developed for visual image classification can be leveraged from an architecture and training method perspective for use in radar applications.
Data Set
In order to help you understand the techniques and code used in this article, a short walk through of the data set is provided in this section. The data set was gathered from radar samples as part of the radar-ml project and found here. This project employs autonomous supervised learning whereby standard camera-based object detection techniques are used to automatically label radar scans of people and objects.
The data set is a Python dict of the form:
{‘samples’: samples, ‘labels’: labels}
samples is a list of N radar projection numpy.array tuple samples in the form:
[(xz_0, yz_0, xy_0), (xz_1, yz_1, xy_1),…,(xz_N, yz_N, xy_N)]
Where a radar projection is the maximum return signal strength of a scanned target object in 3-D space projected to the x, y and z axis. These 2-D representations are typically sparse since a projection occupies a small part of scanned volume. Sampling, storing and making use of the 2-D projections can be more efficient than using the 3-D source data directly.
labels is a list of N numpy.array class labels corresponding to each radar projection sample of the form:
[class_label_0, class_label_1,…,class_label_N]
Projections from a typical single sample are shown in the heat map visualization below. Red indicates where the return signal is strongest.
This data was captured in my house in various locations designed to maximize the variation in detected objects (currently only people, dogs and cats), distance and angle from the radar sensor.
The data set contains only a few thousand samples (with known labeling errors) and can only be used to train a deep neural network for a small number of epochs before over fitting. The labeling error will affect the accuracy of the radar classifier trained from this data set. While a future effort will attempt to fine-tune the object detector to reduce the error, using the SGAN may obviate or minimize the need to label future radar observations. Reducing the number of labeled data points to train a classifier, while maintaining acceptable accuracy, was the primary motivation to explore using SGANs in this project.
Model Architectures
As noted above, the data set contains a collection of 2-D representations of a 3-D radar image and fortunately, there is prior work from the computer vision world in demonstrating its possible to design and train neural networks on such 2-D representations that match or even outperform networks trained on the native 3-D data set, see [2] and [3]. This prior work inspired the development of the networks below.
All models and associated training were implemented using the Keras API, the high-level API of TensorFlow as part of the radar-ml project.
Deep Neural Network (DNN) Classifier
Although not recognizable by a human, the collection of 2-D radar image projections contain features that map back to the scanned object. Each of the three 2-D projections are passed through separate 2-D convolution layers that learn these features and successively down-sample the image. The output from these layers are concatenated and then flattened to form a single feature vector which is used as an input to deeply connected dense layers followed by a classification layer. This architecture in the figure below. You may notice that a single branch of this architecture is similar to a Convolutional Neural Network (CNN) used in computer vision.
The model is implemented by the Python module in the file dnn.py in the radar-ml repository. You can see the code snippet that defines and compiles the model below.
Semi-Supervised Generative Adversarial Network (SGAN)
Gathering radar images for model training is relatively straightforward compared to establishing ground truth which requires a human in the loop, autonomous supervised learning, or a technique such as Semi-Supervised learning that combines a small amount of labeled data with a large amount of unlabeled data during training. The motivation to use Semi-Supervised learning was to minimize the effort associated with humans labeling radar scans or the use of complex (and, possibly error prone) autonomous supervised learning.
The Generative Adversarial Network (GAN) is an architecture that uses unlabeled data sets to train an image generator model in conjunction with an image discriminator model. In some cases you can use the discriminator model to develop a classifier model. GANs have been used in radar signal generation [4] and have found extensive use in computer vision applications [5].
The Semi-Supervised GAN (SGAN) model is an extension of a GAN architecture that employs co-training of a supervised discriminator, unsupervised discriminator, and a generator model. In this project, the supervised discriminator is used as a classification model that generalizes to novel data sets and a generator model that yields realistic examples of radar projections (used only as a validity check).
The supervised discriminator architecture is shown in the figure below and you may notice its similar to the DNN architecture shown nearby, with some exceptions including the use of LeakyReLU (Leaky Rectified Linear Unit) instead of ReLU which is a GAN training best practice [7]. The model includes Batch Normalization layers to aid training convergence which is often a problem in training GANs [6]. The unsupervised discriminator shares most layers except for the final output layers and so has a very similar architecture. The supervised discriminator’s output is a dense layer with softmax activation that forms a 3-class classifier while the unsupervised model takes the output of the supervised model prior to the softmax activation, then calculates a normalized sum of the exponential outputs [6].
Both the supervised and unsupervised discriminator models are implemented by the Python module in the file sgan.py in the radar-ml repository. A code snippet that defines and compiles the model below.
The generator model takes a vector from the latent space (a noise vector drawn from a standard Normal distribution) and uses three branches of transposed convolution layers with ReLU activation to successively up-sample the latent space vector to form each of the three radar image projections. The generator is stacked on top on the discriminator model and is trained with the latter’s weights frozen. This combined architecture is depicted in the figure below. Note the use of Batch Normalization layers to aid model training convergence.
The generator and GAN are implemented by the Python module in the file sgan.py in the radar-ml repository. Below is a code snippet that defines and compiles the model.
Model Training and Results
DNN
The DNN is trained via the tf.keras.Model class fit method and is implemented by the Python module in the file dnn.py in the radar-ml repository. Below is a code snippet of the training function not shown are the steps required to pre-process and filter the data.
The results from a typical training run are below. The current state of the model and data set is capable of obtaining validation set accuracy in the mid to high 80%’s. More work is required to match or exceed the ~ 90% accuracy obtained by SVM and Logistic Regression models in previous work [8][9]. This will be the focus of future effort.
SGAN
Generative Adversarial Networks, or GANs, are challenging to train. This is because the architecture involves both a generator and a discriminator model that compete in a zero-sum game. It means that improvements to one model come at the cost of a degrading of performance in the other model. The result is a very unstable training process that can often lead to failure, e.g. a generator that generates the same image all the time or generates nonsense. As such, there are a number of heuristics or best practices (called GAN hacks) that can be used when configuring and training your GAN models. These heuristics have been hard won by practitioners testing and evaluating hundreds or thousands of combinations of configuration operations on a range of problems over many years.
“Generative Adversarial Networks with Python”, Jason Brownlee, 2021.
You can find many good papers and articles that can help to understand how to apply best practices for training GANs. In particular, Jason Brownlee has published many pragmatic articles and papers that can prove time-saving [7]. Some of this work was used to determine a training method that worked reasonably well on the radar SGAN models and data set. Consider reading his online articles and buying his e-books if you are serious about understanding and applying machine learning.
The training loop is implemented by the Python module in the file sgan.py in the radar-ml repository. Below is a snippet of the training loop, not shown are the steps required to pre-process and filter the data set as well as several helper functions. This code is based on reference [7]. Note that the discriminator model gets updated with 1.5 batches worth of samples but the generator model is updated with one batch worth of samples each iteration.
You will find the training results to vary from run to run due to the stochastic nature of GANs, so its best to average results over several runs. A good training session will have moderate (~ 0.5) and relatively stable losses for the unsupervised discriminator and generator while the supervised discriminator will converge to a very low loss (< 0.1) with high accuracy (> 95%) on the training set. Accuracy results on the validation set tends to be in the low to high 70%’s with losses hovering around 1.2 with using only 50 supervised samples per class. This is an encouraging result but clearly more modeling work and data collection is required to get the validation accuracy on par with the other machine learning methods that were employed on this data set, which were typically ~ 90% [8][9]. This will be the focus of future work on this project. Typical training results are shown below.
You should also view the images produced by the generator to determine if they make sense. In this case, since the images are 2-D projections of radar scans of 3-D objects and are not recognizable by a human, the generated images need to be compared to examples from the original data set like the one above. The figure below is a set of generated 2-D scans. A similarity in one of the projections (the X-Y plane) is evident but not obvious in the others, at least for this training run. This could account for the low accuracy and finding ways to make the other generated projections visually similar to the training set is left to a future exercise.
Conclusions
You can leverage model architectures from CNN’s, SGAN’s and associated training techniques developed for camera-based computer vision to develop neural networks to classify radar images. Given the dearth of radar data sets, you are typically required to collect radar data sets which can be resource intensive and error-prone to ground truth novel radar observations. You can use self-supervised techniques to make use of unlabeled data using only a few tens or less of labeled samples per class and an SGAN. In this manner, you can feasibly develop radar image classifiers using large amounts of unlabeled data.
Previous work used shallow machine learning models and achieved higher accuracy on the data set than currently obtained using the networks and techniques described here. Future efforts are planned to close this gap and to increase the size of the data set to obtain better validation set accuracy before over fitting.
References
- Semi-Supervised Learning with Generative Adversarial Networks, 2016.
- Multi-view Convolutional Neural Networks for 3D Shape Recognition, 2015.
- Multi-view classification with convolutional neural networks, 2021.
- Generative Adversarial Network for Radar Signal
Generation, 2019. - Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2016.
- Improved Techniques for Training GANs, 2016.
- Generative Adversarial Networks with Python, 2021.
- Teaching Radar to Understand the Home, 2020.
- Using Stochastic Gradient Descent to Train Linear Classifiers, 2020.
