Building A Realtime Pothole Detection System Using Machine Learning and Computer Vision

Figure 1: Screenshot of potholes detected by a camera installed on a moving vehicle. The potholes are marked with rectangular regions with detection confidence (image by author)

Potholes are formed due to wear and tear and weathering of roads. They cause not only discomforts to citizens but also deaths due vehicle accidents. The US records more than 2000 fatal accidents per year due to potholes and bad road conditions.

This article describes an approach of building a realtime detection system.

There are numerous use cases of this detection system. For example, a civic authority can detect and locate potholes and assess their magnitudes so that they can plan for repairs. Cameras installed on moving vehicles can detect potholes in realtime and help drivers avoid potholes.

At a very high level, the steps are as follows:

1. Data acquisition and preparation
2. Model training and evaluation
3. Model deployment for realtime detection

1. Data preparation

We need labeled images for a supervised machine learning model. We collected 1000s of pothole images from the Internet and labeled them using an image annotation tool. There are some good image labeling tools out there — both commercial and open source ones. We used VoTT from Microsoft. This is open source and MIT licensed and available on github, https://github.com/microsoft/VoTT.

Figure 2: Pothole images labeled using VoTT (image by author)

As shown in Figure 2, we loaded images in VoTT and labeled for a single label, e.g. pothole. To learn how to do annotations using VoTT, checkout their official documentation available at the GitHub site.

VoTT allows exporting annotated images in different formats. We exported the images in the TensorFlow compatible format called TFRecord.

Training set: We typically take 80% of the labeled images as training set. Store all training TFRecords in a directory, say pothole_training.

Test set: The remaining 20% of the labeled images are set aside as a test set. We store them in a separate directory, say pothole_test.

Machine Learning Model Training

After having all the images labeled and separated into training and test sets, we are ready to train our model. We will train an object detection model using the Single Shot Multibox Detection or SSD algorithm. We are selecting SSD over other object detection models because it offers reasonable accuracy with greater speed compared to other algorithms such as RCNN or its variants. YOLO also offers a great accuracy and speed but for this article we stick to SSD. Other advantage of SSD is that it can detect smaller objects in the image.

We use Momentum AI for training the model. This allows us to train the models and iterate through the modeling process without writing a single line of code. Momentum SSD runs on top of TensorFlow execution engine and provides a scalable platform to run the training both on CPUs and GPUs. Of course, you can write TensorFlow to train your model.

Using its intuitive UI, it is easy to configure and train SSD on Momentum.

As shown in Figure 3 below, click the button that says “Train New Model”, select the “SSD Object Detection” checkbox, fill out the form (the actionable elements are highlighted in yellow for the demonstration purpose).

Figure 3: Screenshot of SSD Model configuration page of Momentum AI (image by author)

A few noteworthy items of configuration:

training_data_dir: this is the directory that contains the TFRecord files for training

test_data_dir: this is the directory that contains the TFRecords for testing.

label_map_file: This is a file that gets generated when you export TFRecords from VoTT.

network_config: Training a model from scratch is a very time consuming and compute intensive task. Momentum provides options to perform transfer learning using the following pretrained checkpoints:

ResNet50/RetinaNet50, input size 640x640
ResNet50/RetinaNet50, input size 1024x1024
ResNet101/RetinaNet101, input size 640x640
ResNet101/RetinaNet101, input size 1024x1024
ResNet152/RetinaNet152, input size 640x640
ResNet152/RetinaNet152, input size 1024x1024
MobileNet, input size 320x320
MobileNet, input size 640x640

Before we run the model, we need to upload the TFRecords (both training and test sets and label_map_file to Momentum. You can upload the files to corresponding directories using Momentum’s “Data Management” utility (as shown in Figure 4 below)

Figure 4: Momentum screenshot showing TFRecords upload using Data Management utility (image by author)

Run the model using Momentum’s UI. For that, expand My Models from the left side menu, click on the Pothole model that you just configured, and click on the “Run” button (highlighted yellow in Figure 5 below for demonstration).

Figure 5: Momentum screenshot- run the SSD model. Screen also shows the buttons to evaluate and deploy the model (image by author).

Figure 5 shows the “Run” button which will trigger the training task when clicked. It will take a while before the model is completely trained. The amount of time the model will take depends on the amount of labeled data, the input image size, batch size, and overall machine capacity.

Momentum executes the training on TensorFlow. It also allows running the model on remote server/cluster where TensorFlow with GPU is installed.

Monitoring Model Execution

While the training is going on, we can monitor the model performance by launching TensorBoard. The following Figure 6 shows Tensorboard screen showing various model losses during the training.

Figure 6: Training losses shown on Tensorboard (image by author)

When the model is fully trained and losses are minimize (as observed on Tensorboard), you can deploy the model on Momentum. Alternatively, you can download the TensorFlow compatible model and write custom code to perform pothole detection.

Sample Output of Realtime Pothole Detection

The following video shows pothole detection by a camera attached to a car moving at 50 miles per hour. This video is the end product developed by a group of Graduate students of George Mason University as part of their Capstone project sponsored by Accure.

Momentum is an enterprise AI platform and available for free for learning and educational purposes. You can signup for a free account at https://impulse.accure.ai/register/ .

Pothole detection was an Accure sponsored Capstone project and I had an opportunity to supervise and mentor a team of Graduate students of George Mason University.