Ultrasound for COVID-19 — A deep learning approach
About an open-access initiative to leverage the automatic detection of COVID-19 from ultrasound
What you will learn in this article:
- CT and X-ray are common assessment methods in the diagnostic process of COVID-19, but the medical community has also advocated ultrasound imaging
- Ultrasound is preferable because it is cheap, portable, easy to sterilize, non-irradiating and available everywhere
- Our deep learning approach demonstrated for the first time that the automatic detection of COVID-19 on ultrasound might be feasible
- With interpretable machine learning methods we take a step towards a decision support tool for doctors
- An open-access data initiative was started on GitHub — Contribute!
https://github.com/jannisborn/covid19_ultrasound
To date, more than 140 million people have been infected by COVID-19, and over three millions have died. The disease is still spreading across the globe, with almost all countries being affected. A key factor in the efforts to keep the virus contained is fast and reliable testing. The COVID-19 RT-PCR test (reverse transcription polymerase chain reaction), which detects viral RNA, is considered as most reliable, but its sensitivity varies widely between countries, sometimes with up to 30% false negatives [1], and it is hardly available in developing countries. Also, PCR tests take a few hours, which is problematic for example in triage situations where doctors have to decide immediately whether to isolate a patient.
This is where medical imaging comes in. The most prominent one to date is CT, which has been used already early in the pandemic for fast diagnosis of COVID-19. Studies have shown that clear distinctive patterns are visible on CT, for example, air space consolidations and so-called (multifocal) ground glass opacities [2]. Judging from the reports, CT seems to be a very promising tool, sometimes even sensitive when a PCR test fails [3]. But there are major flaws: CT is highly radiating, expensive and hard to sterilize. Clearly, these drawbacks impede the extensive use of CT for diagnosis. As an alternative, research has considered X-rays, but the predictive power was shown to be inferior. But there is another medical imaging tool, that — despite its general popularity — has not got much attention in the context of COVID-19. We’re talking about ultrasound.
Point-of-care ultrasound (POCUS) is
- Cheap: While one X-ray examination is estimated to cost around $370, and CT ranging from $500 to $3000, ultrasound (US) is a bargain with only approximately $140. Also, the device itself is cheap and thus easy to distribute, starting from $2000 for portable devices.
- Easy to use: Almost all doctors know how to perform an ultrasound. There are no safety measures as with radiation, and the devices are handy.
- Fast: With one device, it is possible to perform 4 to 5 lung screenings per hour
- Portable: “point-of-care” says it all. The patients do not have to be moved, which saves lots of time and effort.
- Safe: With US, you do not use any irradiating element. Period. Any X-Ray or CT examination slightly increases the lifetime risk of cancer, especially for younger patients.
Despite these advantages, ultrasound has only been integrated into the diagnostic process of lung diseases in the past few years. Certain characteristic pathological patterns, such as so-called B-lines, A-lines and barcode signs, can be analyzed to diagnose so-called pleural effusion, alveolar consolidation, interstitial syndrome and pneumothorax. As COVID-19 alters the lung ultrasound patterns in similar ways, the applicability of ultrasound for COVID-19 was now studied in several publications and its sensitivity was compared to CT [4]. The result is very clear: In accordance with [5,6,7], the authors of [8](published in the prestigious journal The Lancet Respiratory Medicine) advocate for a more prominent role of US for COVID-19 diagnosis, and provide evidence that the sensitivity to detect COVID-19 from US is very similar to the one of CT.
So does this mean that soon ultrasound will replace CT as a diagnostic tool? Unfortunately, it is not that simple. A noteworthy drawback is that doctors have to be trained to recognize COVID in the US and observing COVID-specific patterns is not an easy task but requires some experience. In the current situation, the time for extensive training is limited.
We believe that a solution is given by an assistance system for doctors that automatically classifies ultrasound recordings with computer vision techniques. Such a system could support doctors in their decisions, and give a first assessment of the probability that the patient is infected. In our research, we took the first steps towards this system, collecting a dataset of US recordings from various online sources, pre-processing them and training a neural network to classify images. Our pre-print on arXiv “POCOVID-Net: Automatic Detection of COVID-19 From a New Lung Ultrasound Imaging Dataset (POCUS)” was actually the first work on ultrasound for COVID. In our article published in the Applied Sciences journal “Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis” we then provide a comprehensive study of (interpretable) deep learning methods for classifying lung US data.
Let’s dive in and discuss our work more in detail:
A new POCUS dataset
A main contribution is the collection of a dataset of currently more than 250 recordings (92 COVID-19, 73 bacterial pneumonia and 90 healthy controls). The dataset comprises data from various sources, including unpublished clinical data collected in a hospital by our collaborator in Northumbria (UK), as well as recordings from healthy volunteers scanned in Neuruppin (Germany), but also data from other publications and educational websites. Note that a lot more work was necessary to pre-process the data— in short, we manually cropped the videos and removed artifacts, before selecting frames from the videos to create an image dataset. We even checked the quality of our data with the help of a medical doctor in our team, who could provide valuable comments about visible patterns in each video. Finally, this yielded a clean dataset of more than 3000 images. Of course, this is by no means exhaustive and there is much data out there that needs to be found and processed. However, we consider it a major part of our work that we make the data and pre-processing pipelines available on our GitHub page, where you can contribute with data to our open-access initiative.
A deep learning approach
But is this data useful, can a neural network learn to detect COVID-19 from US images? To answer this question, we trained various neural networks to distinguish between COVID-19, pneumonia and healthy patients. Our best performing model utilises a pre-trained VGG-16 model as a backbone, followed by a single fully connected layer. Augmentation such as rotations, shifts and flips help to prevent overfitting on the dataset that is still rather small for a model with two million trainable parameters.
The results are very promising — our model achieves an accuracy of 88% to classify into COVID-19, pneumonia and healthy patients. Below you can see the confusion matrices, normalized along each axis respectively. The results were obtained in stratified 5-fold cross validation, ensuring that frames from the same video are not in different folds. As visible below, the performance is balanced between classes, meaning that sensitivity and specificity for both COVID-19 and bacterial pneumonia are high. This also shows that deep learning methods might be a worthwhile endeavour not only for COVID-19, but also in general for lung pathologies.
Explainable AI — A decision support system
When we present these results, we are often asked whether we try to replace doctors with such methods. The clear answer is No! Instead, we want to provide a second opinion, a system that can assist doctors in the diagnostic process. For example, wouldn’t it be nice if a software could stop the video at specific time points where you can see something interesting on the ultrasound, and then highlight abnormalities on the image?
One method to do so is called class activation mapping (CAM)[9], which highlights the parts of the image that were most decisive in the prediction of the neural network. We implemented CAM for our classification network, and observe interesting results: Often, the heatmaps focus on so-called consolidations visible on the ultrasound, which indicates fluid in the lung. Also, horizontal “A-lines” on ultrasound that are typical for a healthy lung are often picked up by the CAMs, as shown in the example.
Now to find out whether these activation maps could actually be useful for the clinical practice, we asked two medical doctors to rate fifty examples of activation maps overlayed on a video. The CAMs were rendered useful in general. We therefore believe it could be a very interesting research direction to improve the detection of specific pathological patterns and to leverage the implementation of an actual software that could be used by doctors.
What’s next?
So how to transform this preliminary but promising work into an actual application? It’s a long way to go, but for this, we need your help. As a first step, we need more data to validate our model — If you have any data that can be useful for our purposes: Contribute to our dataset via our GitHub page. We are also eager to support any work aiming to improve the machine learning models with our dataset.
With these initial results, a major step is also to evaluate the approach on real clinical data. For this purpose we are currently collaborating with a hospital to conduct a clinical study to validate of our approach. A further study could systematically compare CT, X-Ray and ultrasound and their respective automatic detection approaches in a clinical study. This would be a great leap forward.
But if you do not know any data sources and are no machine learning expert, there are still many possibilities to contribute. A very simple thing to do is to raise awareness for the possibility of using POCUS as a diagnosis tool. Share our publications, watch our promo video or check out our devpost, and visit our GitHub landing page where all code is available. Get in touch if you want to contribute or have questions, and any feedback is welcome. Thanks for reading about our work!
References
[1] Yang, Yang, et al., “Laboratory diagnosis and monitoring the viral shedding of 2019-nCoV infections.”(2020) MedRxiv.
[2] Jeffrey P Kanne. Chest ct findings in 2019 novel coronavirus (2019-ncov) infections from wuhan, china (2020) key points for the radiologist.
[3] Ai, Tao, et al. “Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases.” (2020) Radiology: 200642.
[4] Fiala, M. J. “Ultrasound in COVID-19: a timeline of ultrasound findings in relation to CT.”(2020) Clinical Radiology.
[5] Smith, M. J., et al. “Point‐of‐care lung ultrasound in patients with COVID‐19–a narrative review.” (2020) Anaesthesia.
[6] Sofia, Soccorsa, et al. “Thoracic ultrasound and SARS-COVID-19: a pictorial essay.” (2020) Journal of Ultrasound: 1–5.
[7] Soldati, Gino, et al. “Is there a role for lung ultrasound during the COVID‐19 pandemic?.”(2020) Journal of Ultrasound in Medicine.
[8] Buonsenso, Danilo, Davide Pata, and Antonio Chiaretti. “COVID-19 outbreak: less stethoscope, more ultrasound.” (2020) The Lancet Respiratory Medicine.
[9] Zhou, Bolei, et al. “Learning deep features for discriminative localization.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
