Image Processing with Python — Using RG Chromaticity
How to use the Gaussian Distribution for Image Segmentation
Segmenting images by their color is an extremely useful skill to have. I’ve previously written an article on how to do this via the RGB and HSV color spaces. However, another effective way to segment images based on their color is by making use of RG Chromaticity and the Gaussian Distribution. This article will discuss how to do just that.
Let’s begin!
As always, first import the required Python libraries.
import numpy as np
import matplotlib.pyplot as plt
from skimage.io import imread, imshow
from numpy import ndarray
from matplotlib.patches import Rectanglefrom mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib import colors
Excellent, now let us import the image we shall be working with.
budapest = imread('budapest.png')
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(budapest);
Some of you may be familiar with the above building, it is a shot of the the Hungarian parliament I took some years ago. As it is quite a colorful image, it is ideal for our article.
I’m sure most of us are aware that theimages rendered on our screens are actually in the RGB color space. A color space that uses different combinations of Red, Green, and Blue to generate a spectrum of different colors. To help us visualize, let us split the image into its different components.
def rgb_splitter(image):
rgb_list = ['Reds','Greens','Blues']
fig, ax = plt.subplots(1, 3, figsize=(17,7), sharey = True)
for i in range(3):
ax[i].imshow(image[:,:,i], cmap = rgb_list[i])
ax[i].set_title(rgb_list[i], fontsize = 22)
ax[i].axis('off')
fig.tight_layout()rgb_splitter(budapest)
Essentially, if we isolate each color this would be the result. Note that just because a certain color is present in a pixel, does not necessarily mean that the pixel will appear as yellow to our human eyes. Remember that what our screens merely simulate the colors and do not generate the colors themselves.
Now let us discuss RG Chromaticity.
RG Chromaticity is merely a 2 dimensional normalize version of the RGB color space. We can convert our image to this color space by dividing each color channel’s value by the summed value of that specific pixel. From a programming perspective it would take the form below.
budapest_r = budapest[:,:,0] /budapest.sum(axis=2)
budapest_g = budapest[:,:,1] /budapest.sum(axis=2)You may notice that we did not include the Blue channel, this is by design. We know that no matter what, the sum of all the normalized pixels should be equal to 1. Hence, the Blue channel can be derived from the other two. The below code does just that.
one_matrix = np.ones_like(float,shape=budapest_r.shape)
budapest_b = one_matrix- (budapest_r +budapest_g)Now you may be asking yourself, “What does this image look like?”. Well let us see what happened to the image.
Note that in our image white is actually represented by black and that the luma or brightness is the same across all pixels. You may have a difficult time actually seeing the image at this point, this is because the human eye is better suited to differentiating objects based on light contrast (but I’m not an ophthalmologist so please don’t ask me the details).
Of course whether we can actually visualize the image or not does not matter, the true strength of RG Chromaticity lies in being able to represent all the pixel values in a 2-Axis graph (remember the Blue channel can be derived).
Let us plot out how our image can be represented on such a graph.
def RG_Chroma_plotter(red,green):
p_color = [(r, g, 1-r-g) for r,g in
zip(red.flatten(),green.flatten())]
norm = colors.Normalize(vmin=0,vmax=1.)
norm.autoscale(p_color)
p_color = norm(p_color).tolist() fig = plt.figure(figsize=(10, 7), dpi=100)
ax = fig.add_subplot(111) ax.scatter(red.flatten(),
green.flatten(),
c = p_color, alpha = 0.40) ax.set_xlabel('Red Channel', fontsize = 20)
ax.set_ylabel('Green Channel', fontsize = 20) ax.set_xlim([0, 1])
ax.set_ylim([0, 1]) plt.show()RG_Chroma_plotter(budapest_r,budapest_g)
The above chart gives us a nice visual representation of the different pixels present in our image. Now, how can we make use of this knowledge to properly segment colors.
what we need to do first is select a patch from the image.
patch = budapest[500:510,50:60]
imshow(patch);
Let us see where our patch is located on the RG chromaticity graph.
patch_r = patch[:,:,0] /patch.sum(axis=2)
patch_g = patch[:,:,1] /patch.sum(axis=2)
RG_Chroma_plotter(patch_r,patch_g)
What we will do is essentially create a mask that will use the patch’s properties as an input to create a gaussian distribution. What this does is locate where the patch’s pixel lie on the original image and use that information to create a mask.
The below is a helpful function that does just that for us.
def gaussian(p,mean,std):
return np.exp(-(p-mean)**2/(2*std**2))*(1/(std*((2*np.pi)**0.5)))def rg_chroma_patch(image, patch_coor, mean = 1, std = 1):patch = image[patch_coor[0]:patch_coor[1],
patch_coor[2]:patch_coor[3]]
image_r = image[:,:,0] /image.sum(axis=2)
image_g = image[:,:,1] /image.sum(axis=2)
patch_r = patch[:,:,0] / patch.sum(axis=2)
patch_g = patch[:,:,1] / patch.sum(axis=2)
std_patch_r = np.std(patch_r.flatten())
mean_patch_r = np.mean(patch_r.flatten()) std_patch_g = np.std(patch_g.flatten())
mean_patch_g = np.mean(patch_g.flatten()) masked_image_r = gaussian(image_r, mean_patch_r, std_patch_r)
masked_image_g = gaussian(image_g, mean_patch_g, std_patch_g)
final_mask = masked_image_r * masked_image_g fig, ax = plt.subplots(1,2, figsize=(15,7))
ax[0].imshow(image)
ax[0].add_patch(Rectangle((patch_coor[2], patch_coor[0]),
patch_coor[1] - patch_coor[0],
patch_coor[3] - patch_coor[2],
linewidth=2,
edgecolor='b', facecolor='none')); ax[0].set_title('Original Image with Patch', fontsize = 22)
ax[0].set_axis_off()
#clean the mask using area_opening
ax[1].imshow(final_mask, cmap = 'hot');
ax[1].set_title('Mask', fontsize = 22)
ax[1].set_axis_off()
fig.tight_layout()
return final_maskfinal_mask = rg_chroma_patch(budapest, [500,510,50,60])
We can see that the function does a pretty decent job at creating a mask. To validate, let us apply the mask to our image and see the result. First let us binarize our mask.
binarized_mask = final_mask > final_mask.mean()
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(binarized_mask)
Excellent, the below function will now take the original image and the binarized mask and give us the final result.
def apply_mask(image,mask):
yuv_image = rgb2yuv(image)
yuv_image[:,:,0] = yuv_image[:,:,0] * mask
masked_image = yuv2rgb(yuv_image)
fig, ax = plt.subplots(1,2, figsize=(15,7))
ax[0].imshow(image)
ax[0].set_title('Original Image', fontsize = 22)
ax[0].set_axis_off()
ax[1].imshow(masked_image);
ax[1].set_title('Masked Image', fontsize = 22)
ax[1].set_axis_off()
fig.tight_layout()
We can see that the mask does an excellent job. Everything except the field is blackened. Note how we made use of the Y’UV color space, this is because the Y’UV color space has a channel dedicated to brightness.
For fun let us apply all these to another image and see if we can equally impressive results.
singapore = imread('singapore_street.png')
plt.figure(num=None, figsize=(8, 6), dpi=80)
We shall utilize the above image of a street in Singapore. First let us see what the image’s RG Chromaticity chart looks like.
singapore_r = singapore[:,:,0] /singapore.sum(axis=2)
singapore_g = singapore[:,:,1] /singapore.sum(axis=2)
RG_Chroma_plotter(singapore_r,singapore_g)
We see that unlike the image of the Hungarian Parliament, the image of Singapore has far more green and blue pixels.
Let us now try to pick out a patch and get the associated mask.
final_mask_singapore = rg_chroma_patch(singapore, [125,150,290,310])
We can see that the mask does a good job at locating the dark pink objects in the image. Let us see what the final masked image looks like.
binarized_mask_singapore = final_mask_singapore >
final_mask_singapore.mean()
apply_mask(singapore,binarized_mask_singapore)
We can see that it does a pretty good job at masking the image as well. Notice how the lanterns are cleanly cut out. However, notice how a lot of the buildings are included as well. In future articles we shall learn how to tune the parameters to ensure that the final image is much cleaner.
In Conclusion
RG Chromaticity is one of the most powerful tools you can use for color segmentation. It’s ability to cleanly mask images is unparalleled by the other techniques we have covered. Going forward will tune the parameters to ensure that we can get cleaner images, but for now I hope that you were able to appreciate this incredibly useful method.
