The Martian Chronicles — When Deep Learning meets Global Collaboration

What you see above are 2 gray-scale photos of the surface of Mars, captured in the vicinity of the landing site of the Spirit, which was a robotic rover built by NASA, active from 2004 to 2010. If you look closely, you will find that the image to the right of the arrow is exactly the same as the one to the left, except that craters have been highlighted in white by a neural network.

If you’re wondering how this remarkable result was achieved, wonder no more and allow me to tell you a tale like none you have ever heard before — one in which you will find out how a bunch of people from various corners of Earth, possessing different skill-sets (related and unrelated to Mars), collaborated together remotely to create an entire pipeline capable of automatically downloading and analyzing raw images of the Martian surface.

I’ve broken this entire 2-month-long “space-trip” into 6 stages and will venture to provide a brief overview of each stage.

The entire space-trip of 2 months, broken up into 6 stages

Origin Story

It all begins with Mars, of course — The second-smallest planet in the Solar System, comprising a thin atmosphere, having surface features reminiscent both of the impact craters of the Moon and the valleys, deserts, and polar ice caps of Earth.

Examples of the Mars’ diverse surface

Recently the search for Technosignatures — measurable properties that provide scientific evidence of past or present extraterrestrial technology — has gained new interest.

Meet Omdena — a global collaborative platform, which brought together 36 Earthlings from 18 countries for 2 months, to identify anomalies on the surface of Mars, which would help in discovering technosignatures and landing sites. I chanced upon this gem quite by accident, when I was scrolling through my Medium newsfeed and stumbled upon the following life-changing article:

Learn AI Through Collaboration, Not Through Competition.

This article turned up at just the right time, when I had completed quite a few ML courses on platforms such as Udacity, fast.ai, Coursera, etc., and was in search of brand new projects I could apply my newly-acquired skills to. I wasted no time in applying and a few days later, I was extremely thrilled to receive an email confirming my role as an ML engineer in this project, after a brief interview.

Collaboration across the next 2 months would not have been possible had it not been for the existence of marvelous online collaborative tools such as:

• Slack, which we used to converse and brainstorm with each other, with the help of aptly-named #channels.
• Zoom, which we used for our weekly sync-up video-conferences where all participants shared their progress and learnings each week.
• GitHub, which we used to contribute and share all development code.
• Google Colaboratory, which provides free access to GPU-enabled Jupyter notebooks to experiment with different models.
• Google Drive, Google Docs and Google Slides, which helped us keep track of all our data and knowledge in an easy-to-read format and present them to others.

Access

2 weeks went by analyzing the requirements before our very own Python package — mars-ode-data-access — was born. It makes life much easier for those trying to access and download images of the Martian surface. All information regarding its usage can be found here — https://github.com/samiriff/mars-ode-data-access.

Sample usage of the mars-ode-data-access Python package

How does data travel all the way from Mars back to our Jupyter notebooks? To cut a long story short, the Mars Reconaissance Orbiter (MRO), built by NASA, captures high-resolution photos of the Martian surface using the High Resolution Imaging Science Experiment (HiRISE) camera and relays them back to Earth via the Deep Space Network (DSN). Common users like us access all this data by opening the Mars Orbital Data Explorer (ODE) Website in our browsers, or better still, by programmatically accessing it using our very-own python package.

If you’d like to dive deeper into the details of this journey, head over to the following article to learn more:

A Journey from Martian Orbiters to Terrestrial Neural Networks

Data

Once we obtained the data, we needed to understand the anomalies we were searching for. Based on the Zenodo dataset, we were able to identify 7 different anomalies, namely craters, dark dunes, slope streaks, bright dunes, impact ejecta, spiders and swiss cheese.

Apart from that, we also identified technosignatures from different landing sites of Mars Missions, such as NASA’s Curiosity Rover’s parachute.

Furthermore, we created pre-processing utilities to align, crop, translate, rotate, scale and transform the downloaded data.

You can find more details about these tools in the following article, written by Sebastian Laverde:

MarsiAn-omaly Detection Through Deep Adversarial Training

Labels

With all our data in place, we had to manually label all images. Gone are the days of simple image classification. We now live in a world of object detection and semantic segmentation, with a variety of annotation tools and labeling formats! Too bad there isn’t a single standard, which is why we had multiple participants using different annotation tools to label different images. Labeling is quite a labour-intensive task and this is when we proved our strength in numbers, accomplishing this seemingly herculean task by labeling 300–400 images, on average, in little more than a week.

We began by using LabelImg — an open source Annotation tool to draw rectangular boxes around anomalies of interest, and also created a utility to automatically convert between PASCAL VOC and YOLO formats.

Drawing bounding boxes over craters with the LabelImg Tool

Then, we used the VGG Image Annotator (VIA) tool to draw polygons around anomalies of interest.

Drawing Polygons around Bright Dunes with the VIA tool

Last but not least, we created a tool — https://github.com/samiriff/image-annotation-converter — capable of automatically converting polygonal annotations from the VIA tool into Segmentation masks with assistance from the LabelMe tool.

Segmentation masks of craters obtained from an image annotated with the VIA tool

Models

Finally, after such elaborate preparations, we reached the stage that everyone had been waiting for — Model selection and training!

We experimented with supervised and unsupervised models, but to keep this story concise, I will be presenting 3 supervised models (SSD, Mask R-CNN and U-Net) and 1 unsupervised model (Ano-GAN). All experiments were run on Google Colaboratory.

Single Shot Multibox Detector (SSD)

With the LabelImg-annotated dataset of 1674 training samples, 350 validation samples and multiple types of image transformations, a Single Shot Multibox Detector backed by a pre-trained Resnet34 model was trained for 30 epochs with a 1-cycle-learning rate scheduler. Focal Loss was selected as the Loss function and was minimized by an Adam optimizer in 6.5 minutes, resulting in the predictions displayed in the image below, with bounding boxes surrounding anomalies.

Parameter Values, Model Architecture and Sample Predictions for an SSD Model

Mask R-CNN

With the VIA-annotated dataset and a higher input resolution of 1024 x 1024 pixels, a Mask R-CNN model backed by a pre-trained Resnet101 backbone, was trained for 30 epochs. The loss function, comprising a combination of bounding box and mask losses, was minimized by an SGD optimizer with momentum in around 3 hours, resulting in the predictions displayed in the image below, with anomalies highlighted in different colors.

Parameter Values, Model Architecture and Sample Predictions for a Mask R-CNN model

U-Net

With the dataset of segmentation masks, a U-Net model backed by a Resnet18 model was trained for 47 epochs with a 1-cycle learning rate scheduler, using different combinations of learning rates and weight decay. The flattened loss of Cross-Entropy Loss function was minimized by an Adam optimizer in around 2 hours, resulting in the predictions in the image below, with anomalies highlighted in white.

Parameter Values, Model Architecture and Sample Predictions for a U-Net Model

Ano-GAN

Last but not least, with a completely unlabelled dataset of 10000 images, a Generative Adversarial Network was trained for 200 epochs. The L1 Reconstruction loss was minimized by an Adam optimizer in around 6 hours, resulting in the predictions displayed on the right. Technosignatures such as heat shields, parachutes, crash sites, and the Curiosity Rover were detected.

Parameter Values, Model Architecture and Sample Predictions for an Ano-GAN model

Read more about this model’s architecture and how it was trained, in this article by Murli Sivashanmugam:

Anomaly detection in Martian Surface — Searching for needles in a haystack!

Before moving further, I would like to take a moment to thank Jeremy Howard and the folks behind fast.ai, for creating such a fantastic library and amazing video lectures, which made it so much easier for us to understand the internal workings of the SSD, U-Net and GAN models. If you haven’t seen this course before, I urge you to visit https://course.fast.ai immediately.

Results

Precision, Recall, and F1-Scores for all label classes

It was found that the U-Net model yielded the best scores. The precision scores for all anomalies are above 90% but there is still room for improvement with regard to the recall scores for technosignatures, slope streaks, and spiders.

The confusion matrix based on Intersection-over-Union (IOU) scores is shown below:

Wait! This story isn’t over yet. Before I end, let me present to you a user-friendly application that was created by tying all these 6 parts together, to create an end-to-end pipeline capable of downloading images directly from the Mars Orbital Data Explorer and analyzing them using our trained U-Net model (Make sure you run this notebook in a GPU-enabled environment):

Google Colaboratory

If you wish to view all our experiments or contribute as well, please visit https://github.com/cienciaydatos/ai-challenge-mars .

Conclusion

So, there you have it — our journey over the past 2 months. I hope you enjoyed reading about our project as much as we enjoyed working on it.

Working collaboratively on this project was akin to training a neural network, wherein things were chaotic at the beginning of the project with random strangers joining a newly-created collaborative environment (similar to how weights and biases in a neural network are initialized randomly during creation) and as time went by, we gradually got to know and learn from each other, refining our understanding and becoming less chaotic by constantly iterating (similar to how a neural network gradually adjusts its weights over multiple epochs to minimize a cost function) until we achieved our goal or came close to it (similar to how a neural network eventually produces expected results).

We owe a huge debt of gratitude to Rudradeb Mitra for bringing us all together and mentoring us; Daniel Angerhausen and Patricio Becerra for guiding us with the problem statement; Michael Burkhardt for publicizing our Medium articles and keeping us up-to-date with everything going on at Omdena; Favio Vazquez for helping us get started with an open-source repository on GitHub.

In a world being plagued by greed, hate, and intolerance, Omdena comes as a breath of fresh air to do away with national barriers. This project is a testament to the fact that bringing together a group of strangers from different corners of the Earth, who have never met each other before; transcending geographical borders and time zones to work together and solve fascinating social problems; whilst learning from and inspiring each other every single day, is not just a pipe dream, thanks to online education, collaborative tools and platforms like Omdena!

If you wish to learn more about Omdena and apply your skills to new collaborative challenges, please do visit https://omdena.com .

Ground Control to Major Tom. Over and out…

References

[1] http://course18.fast.ai
[2] https://zenodo.org/record/2538136#.XTHh3OgzbIV
[3] https://medium.com/omdena/a-journey-from-martian-orbiters-to-terrestrial-neural-networks-93541169f665
[4] https://github.com/cienciaydatos/ai-challenge-mars
[5] https://medium.com/omdena/pushing-the-limits-of-open-source-data-enhancing-satellite-imagery-through-deep-learning-9d8a3bbc0e0a
[6] https://medium.com/omdena/anomaly-detection-in-martian-surface-searching-for-needles-in-a-haystack-169fb0246da7
[7] https://www.jeremyjordan.me/evaluating-image-segmentation-models/
[8] https://arxiv.org/pdf/1905.05055.pdf
[9] https://towardsdatascience.com/an-end-to-end-introduction-to-gans-bf253f1fa52f
[10] https://github.com/rohitgeo/singleshotdetector
[11] https://towardsdatascience.com/review-retinanet-focal-loss-object-detection-38fba6afabe4
[12] https://gilberttanner.com/blog/fastai-image-segmentation