Hi! If you haven’t read the first part of this short series on Image Segmentation, please do so here.
We have discussed thresholding and color segmentation over at my previous story so our part 2 will focus on Chromaticity Segmentation, as well as Image Differencing.
Chromaticity Segmentation
Despite the benefits of HSV over RGB, there are still going to be colors that might require our eyes to take an even closer look to see noticeable differences. An example of which would be our skin colors, where different shades of brown might look the same or different, depending on how an individual interprets the color difference. For this scenario, we can look at the Rg Chromaticity space to generate a mask for our object of interest.
For this case, we will be using this painting as reference and determine the RG Chromaticity using the following codes
#library imports
from skimage.io import imread, imshow
import matplotlib.pyplot as plt
import numpy as np
paint = imread('medium/painting.jpg')
paint_R = paint[:,:,0]*1.0/paint.sum(axis=2)
paint_G = paint[:,:,1]*1.0/paint.sum(axis=2)
plt.figure(figsize=(5,5))
plt.scatter(paint_R.flatten(),paint_G.flatten())
plt.xlim(0,1)
plt.ylim(0,1)
The code above will show the distribution of the Red and Green color space via scatterplot which is shown below
After looking at the spread of the data, we can then get a 2D histogram of the color values
plt.figure(figsize=(5,5))
plt.hist2d(paint_R.flatten(), paint_G.flatten(), bins=100,cmap='binary')
plt.xlim(0,1)
plt.ylim(0,1)
From the histogram, we can identify which colors make up our painting. In order to determine the colors and their specific locations on the histogram, we need to take a reference patch from the image and take its corresponding RG chromaticity.
Let’s try to get the body of water as the patch
patch = paint[150:200,400:450,:]
imshow(patch)
Taking the RG chromaticity is done using the following code
#scatterplot
patch_R = patch[:,:,0]*1.0/patch.sum(axis=2)
patch_G = patch[:,:,1]*1.0/patch.sum(axis=2)
plt.figure(figsize=(5,5))
plt.scatter(patch_R.flatten(),patch_G.flatten())
plt.xlim(0,1)
plt.ylim(0,1)
#histogram
plt.figure(figsize=(5,5))
plt.hist2d(patch_R.flatten(), patch_G.flatten(), bins=100,cmap='binary')
plt.xlim(0,1)
plt.ylim(0,1)
Now that we have the data distribution of the body of water, let’s try out two types of Segmentation using RG Chromaticity: Parametric and Non-Parametric Segmentation.
- Parametric Segmentation
Parametric segmentation requires transforming each element according to the reference patch and image parameters. To do this in our painting image, we will start by fitting a gaussian probability distribution using the mask generated earlier
#Parametric Magic
std_patch_R = np.std(patch_R.flatten())
mean_patch_R = np.mean(patch_R.flatten())
std_patch_G = np.std(patch_G.flatten())
mean_patch_G = np.mean(patch_G.flatten())
#Making the gaussian work
def gaussian(p,mean,std):
return np.exp(-(p-mean)**2/(2*std**2))*(1/(std*((2*np.pi)**0.5)))
Now let’s check the distribution of the patch that we have made using our codes above
x = np.linspace(0,1)
y = gaussian(x,mean_patch_R,std_patch_R)
plt.plot(x,y)
Let’s test our data out in the RG Chromaticity space
#Visualization of R space and G space
#R space
r_test = np.tile(np.linspace(0,1,64),(64,1))
plt.imshow(r_test)
#G space
g_test = r_test.transpose()
plt.imshow(g_test)
#Test sample R and G space into the gaussian distribution with the generated patch
#R space with patch
test_R = gaussian(r_test,mean_patch_R,std_patch_R)
plt.title('R Space and patch')
plt.imshow(test_R)
#G space with patch
test_G = gaussian(g_test,mean_patch_G,std_patch_G)
plt.title('G Space and patch')
plt.imshow(test_G)
prob_test=test_R * test_G
plt.imshow(prob_test)
The image above shows that the color of the patch has a probability of being part of the painting by using only the R coordinate. Let’s try masking our painting using only the R Chromaticity space
prob_R = gaussian(paint_R,mean_patch_R,std_patch_R)
plt.imshow(prob_R)
Now let’s try it out using the G Chromaticity space
prob_G = gaussian(paint_G,mean_patch_G,std_patch_G)
plt.imshow(prob_G)
Multiplying the R and G space gives us the following output
prob=prob_R * prob_G
plt.imshow(prob)
The combined images now indicate that there is an actual probability of the taken color being part of the reference patch that we have taken earlier. We can now segment the image properly by generating a new mask with a given set of probability values
prob_list = [0.2, 0.4, 0.6, 0.8]
fig, ax = plt.subplots(1,4, figsize=(10,10))
for i in range(4):
ax[i].imshow(prob > prob_list[i], cmap='gray')
ax[i].set_title(prob_list[i])
plt.show()
Integrating the mask at our original RGB picture isolates the body of water. For this example, we will use 0.8 as the threshold
mask = prob > 0.8
red = paint[:,:,0]*mask
green = paint[:,:,1]*mask
blue = paint[:,:,2]*mask
paint_masked = np.dstack((red,green,blue))
imshow(paint_masked)
2. Non-Parametric Segmentation
For the cases where the regions of interest are not easily approximated by our generated 2D Gaussian function, a non-parametric approach can be done instead. This works by using the 2D histogram from the painting and using histogram back-projection to mask the image with the computed histogram from the reference patch.
Now that’s been said, we can simply do all that with cv2! Here are some lines of code to accomplish just that. For this example though, we will only use a 1-D histogram, taking into account the Hue from the HSV color space
import cv2 as cv
#histogram and backprojection
def Hist_and_Backproj(val):
bins = val
histSize = max(bins, 2)
ranges = [0, 180] # hue_range
hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)
backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)
cv.imshow('BackProj', backproj)
w = 400
h = 400
bin_w = int(round(w / histSize))
histImg = np.zeros((h, w, 3), dtype=np.uint8)
for i in range(bins):
cv.rectangle(histImg, (i*bin_w, h), ( (i+1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255), cv.FILLED)
cv.imshow('Histogram', histImg)
#getting image and transforming to hsv
src = cv.imread('medium/painting.jpg')
hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)
ch = (0, 0)
hue = np.empty(hsv.shape, hsv.dtype)
#using only 1-D histogram (Hue)
cv.mixChannels([hsv], [hue], ch)
#creating trackbar to change bin values
window_image = 'Source image'
cv.namedWindow(window_image)
bins = 25
cv.createTrackbar('* Hue bins: ', window_image, bins, 180, Hist_and_Backproj )
Hist_and_Backproj(bins)
#show image and allow user to close the program
cv.imshow(window_image, src)
cv.waitKey()
Image Differencing
Now this is a fun subject to talk about, especially with kids. Image Differencing is when instead of looking at color spaces, we want to determine changes or movements in our videos or images, pretty much like how kids play "spot the difference" when shown 2 almost identical pictures.
image1 = imread('medium/19a.jpg')
image2 = imread('medium/19b.jpg')
fig, ax = plt.subplots(1,2,dpi=100)
ax[0].imshow(image1)
ax[1].imshow(image2)
One way to go about image differencing is to simply transform both images into grayscale, and literally subtract both images from one another. The resulting output gives out the changes between the two images
#grayscale
from skimage.color import rgb2gray
image1_gray = rgb2gray(image1)
image2_gray = rgb2gray(image2)
fig, ax = plt.subplots(1,2,dpi=100)
ax[0].imshow(image1_gray,cmap='gray')
ax[1].imshow(image2_gray,cmap='gray')
Now let’s spot the difference
diff = image1_gray - image2_gray
imshow(diff)
And there you have it! Chromaticity Segmentation and Image Differencing
Interested in my work? You can see more stories over at my profile