Data for Change
Creating an AI to Combat Environmental Pollution
How group of Polish women built AI for trash detection in the local environment
The plastic problem
Waste pollution is one of the biggest environmental issues in the modern world.
Based on a study carried out by the University of Newcastle, humans eat up to 250 grams of microplastics each year. That’s two and a half bars of chocolate a year. But a plastic one.
Almost 90 percent of this plastic comes from bottled and tap water. Today, more than 300 million tons of plastic is produced annually and only 30 years left for the amount of garbage in the ocean to exceed the amount of sea creatures.
A group of women from the local Polish chapter of Women in Machine Learning & Data Science (WiMLDS) organization not only thought about the problem but also decided to create a solution, on their own, to be able to detect litter in the environment — using AI.
Using AI to fight the litter
Detect waste in Pomerania was a non-profit, educational, eco project that aimed to use Artificial Intelligence for the general good.
The idea was to use machine learning to detect plastic waste in the environment, educate and support career growth of women from the local community in the field of Data Science and Machine Learning.
The project aimed at people from Pomerania, with at least intermediate Python knowledge and a basic experience in Deep Learning.
For 5 months a team of carefully selected members, 9 female data scientists, analysts and machine learning engineers supported by 5 industry mentors was studying and working together on developing a model for trash detection, a solution that would be applicable for video and photography.
Transfer Learning for object detection
In Detect Waste in Pomerania project we used 9 publicity available datasets, and some data collected using Google Images Download. We used a technique called transfer learning, a machine learning technique where a model trained on one task is re-purposed on a second related task. In our case we use models pretrained on a well-known COCO dataset.
During the 5 months of work we have tried many models and approaches, among them:
- EfficientDet,
- DETR (DEtection TRansformer),
- Mask R-CNN,
- Faster R-CNN,
- Classification with ResNet50,
- Classification with EfficientNet.
But waste detection is not an easy task. It is very easy to overtrain the model and sometimes it is very hard to define what a litter is.
A shoe that I wear is a shoe and not a litter. But if I throw it away in the forest, it will become trash.
Another problem comes with bio litter; small pieces of apple, half eaten tomatoes: it is hard for a neural network to learn them and even harder to distinguish from nature. After all, a fruit under a tree is just a fruit that could fall down from a tree. But it could be thrown by someone and then become litter.
However, we have managed to develop an open-source AI-based solution to litter detection and detect litter with around 65% average precision, classifying it into main categories:
- metals and plastic,
- paper,
- glass,
- bio,
- non-recyclable,
- other unknown.
But, how have we done it? We decided to split the task into two, object localization and object classification.
The first step — localization of waste
We tried to detect waste from high- and low-quality images taken from many public datasets. For this purpose, we used EfficientDet written in PyTorch, a popular state-of-the-art family of object detection models that came out of the Google Brain team. Our main goal was to localize garbage from photos taken in various environmental conditions.
We started with waste detection performed on the TACO dataset — an open image dataset of waste in the environment. At the beginning TACO categories were converted to detect waste categories according to the obligatory segregation rules of the city of Gdańsk. After first attempts, we have noticed that our model performs quite well when localizing waste but works poorly with classification. Therefore, we have decided to try single class detection instead, and separate classification.
In our GitHub repository you will find EfficientDet code already adjusted for our mixed dataset. In our case EffcientDet-D2 trained for 1 litter class gave us the best possible results with almost mAP@0.5 equal 64%. Smaller architectures got lower scores, whereas bigger ones quickly overfitted. Probably due to highly imbalanced data.
Next step — garbage classification
For the classification task we used the EfficientNet-B2 network, which also is a backbone in our best waste detector — EfficientDet-D2. EfficientNet, used in our waste classification challenge, is a Convolutional Neural Network based on AutoML and Compound Scaling.
Additionally, we used multiple data augmentation techniques, which are very important in the waste classification task. The instances of litter can assume different appearances depending on the environment in which they are located — in home, underwater or outside. Therefore we decided to apply some methods implemented in the albumentations library.
In the first step we had to prepare images of waste with proper labels. We cut them from our train and validation subsets of the mix dataset.
The heart of the cropping method is part with reading bounding box properties from annotations, calculating its new coordinates and then cutting from the source image.
Zoom parameter (set to 1 as default) is useful for classification to embed an object in the background, however it can lower the scores if images are crowded with many objects.
Additionally we have used an OpenLitterMap dataset on which we have pre-located the garbage. The biggest difficulty was extracting the object category from the source openlittermap file and assigning it to ours in combination with our predefined localization of waste. Due to missleadning labels we decided to use the data in semi-supervised way.
We also have added eighth class: background (area without any litter, e.g. a sidewalk, a forest path, a lawn), because it was necessary to limit the amount of false positives coming out from our detector.
Generating data for classification purposes could take a while, but in the end in the destination directory we expected to see structure like below.
Our data was loaded to the network in DataLoader with chosen data augmentation techniques (RandomCrop, HorizontalFlip, VerticalFlip, ShiftScaleRotate, RandomBrightnessContrast, Cutout, Normalize) taken from the augmentation library.
When it comes to training the model, we dealt with a set of labeled data (from mixed dataset) as well as a set of unlabeled data (from openlittermap, due to misleading multilabels). In pseudo-labeling concept, the main idea is simple.
- First, train the model on labeled data.
- Then, use the trained model to predict labels on the unlabeled data — make pseudo-labels. Predictions will be treated as a ground-truth label for unlabelled data in the next step.
- Mix pseudo-labels and true training labels and train models with it.
- Repeat previous steps every batch or epoch.
The results
In our case after running 20 epochs of training with a 0.0001 learning rate and batch size of 16 on our neural network with a almost 3/1 train/test split, we achieved a testing accuracy of 73.02% (and 86.67% training accuracy).
The vast majority of our samples from the test subset were classified correctly. The matrix also indicates a high imbalance of the data; unfortunately plastic is the most common litter and, at the same time, the biggest problem when it comes to recycling.
Another problem lies in the recycling rules. There is a lack of clearly defined, uniform guidelines regarding the principles of segregation. It varies from city to city, from country to country.
For example the “chips package” discussed with our annotators team, by someone can be treated as waste belonging to the metals and plastic group, and by others as waste belonging to the non-recyclable group, due to the variety of types of plastic material (not all are recyclable) it is made of.
After all, trash is, unfortunately, everywhere. Left uncollected, it often ends up harming the environment which we are part of. To clean it up efficiently, we need to learn how to recycle it.
If you are interested in the whole solution code of our project, you can find it on our GitHub and you can read more about it on ArXiv.
Authors:
Project and the final solution has been developed by group of WIMLDS Trojmiasto members: Agnieszka Mikołajczyk, Sylwia Majchrowska, Maria Ferlin, Ewa Marczewska, Zuzanna Klawikowska , Marta Plantykow, Magdalena Kortas, Anna Brodecka and Katarzyna Łagocka.
