Road Surface Classification
An approach for road surface type and quality classification
Hello There! In this posting we show an approach and its steps to classify the road surfaces types and quality. And the step-by-step on how to replicate it. In order to test this approach we used the RTK dataset, made by us.
This Dataset [1] consists of images captured with a low-cost camera, with scenarios common in emerging countries, containing unpaved roads and potholes. This is relevant, considering that vehicles with ADAS (Advanced Driver-Assistance Systems) are commercialized in emerging countries, such as Brazil.
The type of pavement is important information for the way a vehicle should be driven, whether by a human or an autonomous vehicle. It involves, in addition to passenger comfort and vehicle maintenance, the safety of everyone involved. We can achieve this with a simple Convolutional Neural Network (CNN) structure from [2].
In this approach we use a specific model to the surface type classification task, which the classes we defined as: asphalt, paved (for all other kinds of pavement) and unpaved. To the surface quality we use other three different models, one for each type of surface. All these four models have the same structure. We take the result from the first model and call the specific quality model.
Before the CNN structure, a Region of Interest (ROI) is defined as a pre-processing step for each input frame. After all, we don’t need the whole image to classify the road. This ROI aims to leave only the part of the image that actually contains road pixels. The top half of the image is discarded, as well as a small portion of the image bottom, because in some frames it may contain part of the vehicle responsible for capturing the images. The ROI is hard-coded because if we use an adaptive ROI it can fails and compromise the model training.
A data augmentation step is performed after this pre-processing. The data augmentation consists of increasing and decreasing the brightness in each frame. This way we improve our training input set and help our system learn to identify the same type and quality road with different illuminations conditions.
Finally, the input images are passed to the CNN structure containing three convolution layers and two fully connected layers.
Road Traversing Knowledge (RTK) dataset
The dataset that was used in this approach, the Road Traversing Knowledge (RTK) [1] was filmed in Brazil, in Águas Mornas and Santo Amaro da Imperatriz cities, Florianópolis neighboring cities, in the Santa Catarina state. The Dataset contains images with different types of surfaces and qualities.
The RTK dataset is available for download at:
http://www.lapix.ufsc.br/pesquisas/projeto-veiculo-autonomo/datasets/?lang=en
Road Surface Type classification
The complete code is available at:
We made use of Python, TensorFlow and OpenCV.
Let’s check it step-by-step…
First, we need to build our surface type classification model. For this, you will need to prepare the data for training the model. You can use the images from the RTK dataset or make your own. The images need to be organized by surface road types.
In our experiments we have used 6264 frames being:
- paved(asphalt): 4344 for asphalt roads.
- paved(concrete): 1337 for different pavements, like cobblestone.
- unpaved: 585 for unpaved, dirt-roads, off-roads.
Next, in train.py, we define from where the training data will be collected. We should separate 20% of the data to be automatically used for validation. We also have defined the batch_size as 32.
The parameters set on train.py will be read on dataset.py class.
At dataset.py class we define the ROI and data augmentation. Two functions for data augmentation, adjust_gamma to decrease the brightness and increase_brightness, with the self explanatory name…
The ROI definition is made for each image when loading the input data.
We also balance the input images, as there are more images of asphalt and less of paved and unpaved roads.
Going back to train.py, let’s define the CNN layers as shown in this TensorFlow tutorial [2]. All images selected to the training step are passed to the first convolution layer, with information about width, height and number of channels. The first two layers have 32 filters with size of 3x3. Followed by a layer with 64 filters with size of 3x3. All the strides were defined as 1 and Padding as 0. A Normal distribution is used for weights initialization. In order to reduce the inputs dimensionally, which helps to analyse features information in the input sub-regions, a max-pooling is applied in all the convolution layers. At the end of each convolution layer, after the max-pooling function, a ReLU is used as an activation function.
After the convolutional layers, the flatten layer is used to transform the convolution multi-dimensional tensor into a one-dimensional tensor.
Two fully connected layers are added at the end. In the first fully connected layer a ReLU activation function is applied. The second fully connected layer has the possible outputs, the desired classes.
We used the softmax function to achieve the probabilities of each class. We also use the Adam optimizer at the end, which update the network weights based in the input data used in the training.
You can train the model running: python train.py in your terminal.
Now, with the model trained we can test. First of all, lets prepare to receive the input test frames and the output filename.
Retrieving the trained model and accessing the graph.
Remember that we do not need the entire image, and our training focused on using a ROI, here we also use it.
Finally, based on the output prediction, we can print the classified surface type in each frame.
You can test the model running in your terminal: python test.py PATH_TO_YOUR_FRAMES_SEQUENCE NAME_YOUR_VIDEO_FILE.avi.
Road Quality classification
Let’s include the quality classification now. We simply use the same CNN architecture used to train the surface type classification model and apply for each class of quality on each surface class separately. So, we trained 3 new models in addition to the existing one. For this, you will need to prepare the data for training the models for each surface class. In the RTK dataset page we already give frames organized by class.
To train each model, run in your terminal:
python trainAsphaltQuality.py
python trainPavedQuality.py
python trainUnpavedQuality.pyNow what changes is in the prediction part. We use four different graphs, one for each model trained.
Restoring the types model
Restoring the asphalt quality model
Restoring the paved quality model
Restoring the unpaved quality model
At this moment, the output prediction take into consideration the quality models as well, we can print the classified surface type and also the quality of this surface in each frame.
Printing the results
You can test running in your terminal: python testRTK.py PATH_TO_YOUR_FRAMES_SEQUENCE NAME_YOUR_VIDEO_FILE.avi.
Some results samples:
Video with the results:
If you have any questions, criticisms or suggestions feel free to reach out. I hope to see you next time. 🤘
This experiment is part of a project on visual perception for vehicular navigation from LAPiX (Image Processing and Computer Graphics Lab).
Acknowledgements
This work was the result of a collaborative effort of a team of engaged researchers:
- Thiago Rateke <rateke.thiago@gmail.com>
- Karla Aparecida Justen <justen.karla@gmail.com>
- Aldo von Wangenheim <aldo.vw@ufsc.br>
If you are going to use the dataset or this approach, please cite as:
@article{rtk:2019,
author = {Thiago Rateke and Karla Aparecida Justen and Aldo von Wangenheim},
title = {Road Surface Classification with Images Captured From Low-cost Cameras — Road Traversing Knowledge (RTK) Dataset},
journal = {Revista de Informática Teórica e Aplicada (RITA)},
year = {2019},
doi = {https://doi.org/10.22456/2175-2745.91522},
}
References
[1] T. Rateke, K. A. Justen and A. von Wangenheim, Road Surface Classification with Images Captured From Low-cost Cameras — Road Traversing Knowledge (RTK) Dataset, (2019), Revista de Informática Teórica e Aplicada (RITA)
[2] A. Sachan, Tensorflow Tutorial 2: image classifier using convolutional neural network, (2017), CV-Tricks.com
