The world’s leading publication for data science, AI, and ML professionals.

COVID-19 CT Analysis using Deep Learning

How we developed a deep learning COVID-19 CT analysis tool with (almost) no data and no annotations

It was mid-January, news about a new virus in china that causes fever and cough started to arrive.

At this time, we were working with a Chinese company, to integrate our chest CT analysis tool. This company developed a cloud PACS (Picture archiving and communication system) that enables radiologists to remotely view cases and edit radiological reports from multiple hospitals.

We noticed patients with lung abnormalities in the chest CT scans that were directed to laboratory tests. As medical imaging algorithm researches, this is when we figured that we could develop a solution to detect such findings.

We were naïve at this time and couldn’t guess that a few months later, the whole world will suffer from the coronavirus pandemic and our Covid-19 CT analysis tool will serve radiologist all over the world as a tool for COVID-19 detection and patient monitoring.

Photo by Dimitri Karastelev on Unsplash
Photo by Dimitri Karastelev on Unsplash

To start with, we faced two problems (which are quite common in the medical imaging world): (1) Small dataset – we had only a few cases of patients suspected for COVID-19; (2) No annotations at all

In the following sections, I will elaborate on how we rapidly built a COVID-19 solution using deep learning tools. The ideas and methods presented here can be used for any new virus or disease with imaging features in CT, especially in the initial phase when data is almost not available.

Deep-Learning framework for COVID-19 chect CT analysis [Image by author]
Deep-Learning framework for COVID-19 chect CT analysis [Image by author]

1. Slice based solution

CT scan include a series of slices (for those who are not familiar with CT read short explanation below). Since we had a very limited number of COVID-19 patient’s scans, we decided to use 2D slices instead of 3D volume of each scan. This allowed us to multiple our data set and to overcome the first obstacle of a small dataset. Another reason for using 2D slices is the ability to be robust to variety of slice thickness of the CT scan (the z margin between two consecutive CT slices).

We gathered 50 abnormal chest CT scans of patients that were diagnosed by a radiologist as suspicious for COVID-19.

Computed tomography (CT) scanning is a diagnostic imaging procedure that uses x-rays to build cross-sectional images ("slices") of the body. Cross-sections are reconstructed from measurements of attenuation coefficients of x-ray beams in the volume of the object studied. Before the data are presented on the screen, the conventional rescaling was made into CT numbers, expressed in Hounsfield Units (HU) [1]. CT slices have high dynamic range (usually 12-bit or 16-bit) and provided in DICOM format.

Chect CT slices [Image by author]
Chect CT slices [Image by author]

code example for reading CT dicom files as numpy array:

2. Lung segmentation

The first step of analysis is to findsegment the lungs in the image, and to crop the image around the lungs. This allows to focus on our region of interest (ROI) for further analysis.

A popular deep-learning architecture for Medical Imaging segmentation tasks is the U-net%e5%92%8c%5bTiramisu%5d(https://arxiv.org/abs/1611.09326.pdf). There are many variations on the original architecture, including the one we used that incorporates a VGG16 pre-trained encoder.

U-net based architecture for lung segmentation [Image by author]
U-net based architecture for lung segmentation [Image by author]

For training, we used 6,150 CT slices of cases with lung abnormalities and their corresponding lung masks. Input slices were clipped using a window of [-1000, 0] HU which represents lung tissue (meaning that all values above 0 are set as 0 and all the values below -1000 are set to -1000). we then normalized the pixels to range [0,1] and resized the image to size 224×224 pixels.

code example for clipping and normalizing CT slice:

We augmented the data using random cropping, horizontal flips and adding normal distributed noise to the lung area to improve the robustness of the segmentation to infectious lungs (which are common for COVID-19 patients).

Adding noise to the lung area as augmentation (noise scale=0.1, noise mean=0.0, noise std=0.8) [Image by author]
Adding noise to the lung area as augmentation (noise scale=0.1, noise mean=0.0, noise std=0.8) [Image by author]

code example for adding normal distributed noise to an image:

Here we used a private dataset, but there are several public datasets of CT scans with corresponding lung masks (LUNA16, Lung CT Segmentation Challenge 2017, StructSeg, MedSeg). MedSeg is for COVID-19 patients.

3. COVID-19 classifier

Since we’ve decided to use 2D slices and due to the fact that we had no annotations to solve a Classification task, we manually annotated each slice of the 50 cases into normal or abnormal, meaning it contains COVID-19 infections (because the infections are visible in sequential slices, it’s not a very complicated annotation task!). Using this approach, we managed to assemble annotated dataset of 1,865 CT slices: 1036 normal and 829 abnormal. To further enrich our training set, we employed data augmentation techniques which included image rotations, horizontal flips, and cropping. Then, we cropped the lungs ROI using the lung segmentation detailed above, and resized each input image to 224X224 pixels to train a COVID-19 classifier per slice, based on a ResNet-50 architecture pre-trained on ImageNet database. Using transfer learning can often improve the classification performance especially where training data is limited. Since the ImageNet classification task includes 1000 categories and we need to classify only 2, we removed the pre-trained model last layers and added two dense layers followed by a sigmoid activation function.

code example for loading a pre-trained ResNet-50 architecture and modify the last layers to fit the new classification task (using Keras):

4. Fine-grained localization

How can you output Localization map with no annotations? Grad-CAM technique is one of the most frequently used interpretation methods in deep learning (designed to visualize where "the network is looking" for classifying specific target). In medical images, this technique is often used for weakly supervised localization and even segmentation, where the supervision is done only by the slice label of normal vs abnormal, to generate localization heatmap images.

Example of heatmap images produced using CAM technique for localization of pathologies in X-ray images. source: Rajpurkar, Pranav, et al. (2017)
Example of heatmap images produced using CAM technique for localization of pathologies in X-ray images. source: Rajpurkar, Pranav, et al. (2017)

While GradCam produces coarse localization maps, we employed the technique over two image resolutions and fused between them to generate fine-grained localization maps of the COVID-19 infected regions. More specifically, we used GradCam on the activation layers corresponding to a resolution of size 14X14 and 28X28 of the ResNet-50 network (defining the GradCam output on the last activation layers of these resolutions).

The two maps were normalized to [0,1] and resized to the input image shape (224X224). We multiplied the two maps to generate a fine-grained localization map which can be applied over the image in color as heatmap image:

Fine-grained localization by fusion of GradCam on two resolutions of images; heatmap image showing high activations on COVID-19 infected region of the lung [Image by author]
Fine-grained localization by fusion of GradCam on two resolutions of images; heatmap image showing high activations on COVID-19 infected region of the lung [Image by author]

5. Case decision and 3D overview

After we produced the lung segmentation and the COVID-19 localization maps for each 2D slice, we can combine these results for all the slices to get the full 3D volume overview of the case. To fuse the localization maps of different slices to a smooth view, we applied thresholding to get the COVID-19 infections segmentation and used 3D gaussian blur.

3D plot image can be generated on the lung and infections binary masks using matplotlib and skimage.measure python libraries:

To conclude the case-level classification we counted the number of slices which the COVID-19 classifier detected as positive and the number of slices belonging to the lung area. If the ratio between them is exceeds a pre-defined threshold (set according to the best performance), the whole case is classified as suspected for COVID-19.

Corona score for patient disease progression monitoring. source: O. Gozes and M Frid-Adar, et al. (2020)
Corona score for patient disease progression monitoring. source: O. Gozes and M Frid-Adar, et al. (2020)

We also suggested a severity score that will allows doctors to follow patient’s COVID-19 disease over time – the "Corona Score". The score is computed by summation over the localization maps of positive detected slices by the COVID-19 classifier, and multiply be the slices pixel spacing (z axis). The Corona Score is a volumetric measurement in units of cm³.

References

[1] https://radiopaedia.org/articles/computed-tomography?lang=us

[2] O. Ronneberger, P. Fischer and T. Brox. U-net: Convolutional networks for biomedical image segmentation%e5%92%8c%5bTiramisu%5d(https://arxiv.org/abs/1611.09326.pdf) (2015). International Conference on Medical image computing and computer-assisted intervention.

[3] M. Frid-Adar, et al. Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pre-trained encoder (2018). Image Analysis for Moving Organ, Breast, and Thoracic Images. Springer, Cham, 2018. 159–168.‏

[4] K. He, et al. Deep residual learning for image recognition (2016). Proceedings of the IEEE conference on computer vision and pattern recognition.

[5] RR. Selvaraju, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization (2017). Proceedings of the IEEE international conference on computer vision.

[6] O. Gozes, et al. Rapid ai development cycle for the coronavirus (covid-19) pandemic: Initial results for automated detection & patient monitoring using deep learning ct image analysis (2020). arXiv preprint arXiv:2003.05037.


The presented method for COVID-19 detection and localization was developed as initial tool of RADLogics where I work as the head of algorithm development.

This is my first Medium blog – I hope you enjoyed! Feel free to write me: [email protected]


Related Articles