SAEMI: Size Analysis of Electron Microscopy Images,
Part 2, How to Obtain a Size Distribution from an EM Image
This article is part 2 in my series detailing the use and development of SAEMI, a web application I created to perform high throughput quantitative analysis of electron microscopy images. Check out part 1 here (where I go over the motivation behind the app) and part 3 here (where I go over how I trained my image segmentation model). You can also check out the app here and its Github here. In this article, I give a walk-through of how to use the app and obtain the best results.
So You Have an Electron Microscopy Image
To start off this article, I am going to assume that you have either seen/taken an electron microscopy (EM) image before or at the very least are familiar with electron microscopes. If not, please check out part 1 in my series where I detail the motivations behind developing this app. Let’s assume, though, that you are a researcher who has taken some EM measurements and now would like to perform some quantitative analysis on your images. More specifically, you would like to determine the mean size of the particles in your image and the standard deviation of that distribution. As an example, let’s say you have an EM image like the one seen in Figure 1 below.
The first thing to note here is the banner that is displayed along the bottom portion of the image which contains information about the measurement such as the scale bar, its electron high tension (EHT) and its magnification. Many electron microscopes and its accompanying software will add this kind of information (at minimum there will be a scale bar) to the image. Unfortunately, since this is part of the image itself, leaving it in may affect the resulting segmentation from the deep learning model.
In order to reduce the potential for error, the additional “meta-information” should be removed from the image whenever possible. Take care to keep a record of the scale bar elsewhere, however, as it will be needed to convert the final calculation from pixels to a physical size in the final step. For now, the “meta-information” can be removed through a number of different methods.
The simplest method and the one I would personally recommend is to just crop it out of the image. It has the least potential to introduce further artifacts and requires the least technical prowess in image processing to perform. The only downside is that you may be losing important data by cropping the image.
In the interest of presenting other options, you can also use some more involved image processing techniques to remove the banner using either the open-cv or scikit-image libraries in Python. Some of these methods include using reflective padding, nearest neighbor padding, and constant padding. An example of all three methods are shown in Figure 2. The region where the banner used to be is highlighted in red in all three examples.
As can be seen, each of these methods can result in unintended artifacts to the image and it is up to you to decide how you would like to deal with the “meta-information”.
Using The App
For now, let’s say you decide to remove the banner by cropping it out. You can then head over to the app using this link and then upload the image using the “Upload Image” button at the top. Alternatively, you can follow the instructions on my github page here to use the app locally on your computer. Either way, once opening the app, you will be greeted with the homepage shown in Figure 3 below. The “Upload Image” button is highlighted in red. In case you don’t have an EM image on hand, you can also use the “Try the Demo” button beside it. This will upload an already prepared demo EM image as an example of how the app works.
Once the image is uploaded to the server, it will be fed through my deep learning model and the size distribution of the particles will be calculated using connected component labeling. The whole process should take about 20–30 seconds. I am continuing to try and speed up the process as much as I can. However, even with such a long running time, it is still much faster than manually labeling EM images that can have hundreds of particles in them at once. When the size distribution has been calculated, you will be greeted with a web-page that looks something like the one shown in Figure 4 below.
Let’s look at each section in more detail.
Size Distribution
The first and probably most important section you will notice is the Size Distribution histogram. This is shown in Figure 5 in more detail.
At the top is the original EM image with the name of the image file displayed above it. The size distribution as calculated by the app is displayed below it in a histogram with the size (in pixels) on the x-axis and the count on the y-axis. Therefore, there are 3 particles that take up approximately 1300 pixels in the image and 1 particle that take up approximately 500 pixels. In the app, you can hover over the data-points to see their exact values.
The mean, median, and standard deviation have also been calculated and are displayed on the upper left of the histogram. Right above the histogram you can also see an input box that says “Enter Bin Size”. This can be used to change the bin size used to define the histogram. The default bin size is 25 pixels. As an example, Figure 6 shows what the histogram would look like with a 75 pixel bin size.
Finally, clicking on “Download Data” at the bottom will download the size distribution as a single column .csv file. You can then perform other calculations such as determining other statistical modes or model the size distribution data however you wish.
The important thing to note, though, is that all of these sizes are calculated in units of pixels ie. how many pixels in the EM image does a particle occupy. In order to obtain a physically relevant measurement, you will have to convert the sizes back to physical units yourself. For instance, in the original image in Figure 1, the scale bar shows that 2 micrometers take up around 110 pixels. Therefore, the 500 pixel size particle actually has an area of approximately 0.9 squared micrometers.
Overlay of Original with Segments
The next section you should check out is the one on the bottom right shown in Figure 7.
This section is meant for you to double check the prediction and ensure that the app was able to correctly segment your EM image. As you can see, the app does a pretty good job of predicting where each of the particles in the image are. For more clarity, you can also use the slider at the bottom of the image to change the opacity of the overlay. If you want the original image to be more clear, you can slide the slider to a lower value. This is shown in Figure 8.
Alternatively, you can slide the slider to a higher value to show the prediction more clearly. This is shown in Figure 9.
Judging from the overlay, it seems the model did a fairly good job of segmenting the particles of the original EM image. However, notice in the bottom left of the segmented prediction, there are two very small predicted particles that do not exist in the original EM image. This is shown in Figure 10 in more detail.
Unfortunately, due to their significantly smaller size, they can skew the calculated mean to a lower value. In this case, the median is probably a better measure of the average particle size in the EM image.
Processing the Resulting Image
In order to ensure the histogram displays as accurate as a result as possible, you can edit out some of the inaccuracies of the prediction using the “Processing Images” section. This is displayed in more detail in Figure 11.
In Figure 11, you can see a couple of different items displayed below the image that will help you edit the prediction. The first two are the undo and redo buttons (represented by left and right arrows respectively) and should be pretty straightforward. Beside them on the right is the save button which will save the changes you made to the image and then recalculate the size distribution based on your changes.
Right below them are two option buttons for “B/W” and “Colour”. If you choose “B/W”, the image displayed above will be only of the binary mask. If you choose “Colour”, the image will actually also show what the app sees as an individual particle. In other words, the image displayed will have unique labels for each connected component in the array. A comparison of the two options are shown in Figure 12.
Displaying the “Colour” option can be useful when you’re unsure whether two very close particles are being considered as one particle or two particles by the app. Keep in mind that neither of these options will edit the image itself. It is only there to provide you with different views of the original prediction.
At the bottom, there is a drop-down menu that allows you to choose how you would like to edit the image. There are three options to choose from: Draw, Remove, and Erase. Choosing these options will change the colour of the brush to white, red, and black respectively.
Editing the image with the “Draw” brush adds whatever you drew onto the prediction. For instance, if there was a particle in the upper left corner that the app had somehow missed, you can fill it in by drawing on the canvas on the bottom left with a white brush. An example of this is shown in Figure 14.
For our image which has spots that were misidentified as particles, however, it is better to use either the Remove (red) or Erase (black) option. By using the red brush, you can remove any particle that has been marked by a red stroke. An example of this is shown in Figure 15.
Using the black brush will simply erase the area from the prediction. While it may seem at first that the red and white brush are the same, they do serve two separate functions. The red brush removes the entire particle while the white brush only removes whichever section has been marked. An example showing the different usages of the two brushes is shown in Figure 16.
Finally, below the two display options there is a slider that will change the “brush” width. An example of what different brush widths look like is shown in Figure 13.
Congratulations! You Have a Size Distribution
After performing the post-processing steps to your satisfaction, you are finally left with a size distribution that accurately reflects your original image. The final histogram is shown in Figure 18.
There is not a great difference compared to the original histogram we obtained in Figure 5 because the model already did a very good job of predicting the particles within the image. As well, there were many particles in the image so even the small mistakes it made were drowned out by the positive particles it identified.
However, if you look closely, you can see the mean has changed from 1.17E3 to 1.23E3 and the standard deviation of the distribution has changed from 5.73E2 to 5.23E2. Once again, you can download this distribution as a .csv file to perform other calculations later. The .csv file contains the sizes that were calculated (in pixels) for all the individual particles that were determined by my deep learning model.
As an example of calculations to perform after downloading the .csv file, you can calculate the mode and the skew of your size distribution if you want more statistical information. You can also aggregate the data from multiple images of the same sample in order to obtain a greater sample size. As well, you can plot multiple histograms that reflect how your sample size distribution changes with respect to time, temperature, pH etc.
If you want to try out SAEMI on your own EM image, you can head over there now and follow the steps outlined in this article. Good luck and please check the final part in this series where I go more in depth into how I trained the deep learning model and what its architecture actually is. Thanks!
