The world’s leading publication for data science, AI, and ML professionals.

Fixation on the Segmentation Part 2: How to do Image Segmentation with Python

Hi! If you haven't read the first part of this short series on image segmentation, please do so here.

Image by Author

Hi! If you haven’t read the first part of this short series on Image Segmentation, please do so here.

We have discussed thresholding and color segmentation over at my previous story so our part 2 will focus on Chromaticity Segmentation, as well as Image Differencing.


Chromaticity Segmentation

Despite the benefits of HSV over RGB, there are still going to be colors that might require our eyes to take an even closer look to see noticeable differences. An example of which would be our skin colors, where different shades of brown might look the same or different, depending on how an individual interprets the color difference. For this scenario, we can look at the Rg Chromaticity space to generate a mask for our object of interest.

For this case, we will be using this painting as reference and determine the RG Chromaticity using the following codes

#library imports
from skimage.io import imread, imshow
import matplotlib.pyplot as plt
import numpy as np
paint = imread('medium/painting.jpg')
paint_R = paint[:,:,0]*1.0/paint.sum(axis=2)
paint_G = paint[:,:,1]*1.0/paint.sum(axis=2)
plt.figure(figsize=(5,5))
plt.scatter(paint_R.flatten(),paint_G.flatten())
plt.xlim(0,1)
plt.ylim(0,1)

The code above will show the distribution of the Red and Green color space via scatterplot which is shown below

After looking at the spread of the data, we can then get a 2D histogram of the color values

plt.figure(figsize=(5,5))
plt.hist2d(paint_R.flatten(), paint_G.flatten(), bins=100,cmap='binary')
plt.xlim(0,1)
plt.ylim(0,1)

From the histogram, we can identify which colors make up our painting. In order to determine the colors and their specific locations on the histogram, we need to take a reference patch from the image and take its corresponding RG chromaticity.

Let’s try to get the body of water as the patch

patch = paint[150:200,400:450,:]
imshow(patch)

Taking the RG chromaticity is done using the following code

#scatterplot
patch_R = patch[:,:,0]*1.0/patch.sum(axis=2)
patch_G = patch[:,:,1]*1.0/patch.sum(axis=2)
plt.figure(figsize=(5,5))
plt.scatter(patch_R.flatten(),patch_G.flatten())
plt.xlim(0,1)
plt.ylim(0,1)
#histogram
plt.figure(figsize=(5,5))
plt.hist2d(patch_R.flatten(), patch_G.flatten(), bins=100,cmap='binary')
plt.xlim(0,1)
plt.ylim(0,1)

Now that we have the data distribution of the body of water, let’s try out two types of Segmentation using RG Chromaticity: Parametric and Non-Parametric Segmentation.

  1. Parametric Segmentation

Parametric segmentation requires transforming each element according to the reference patch and image parameters. To do this in our painting image, we will start by fitting a gaussian probability distribution using the mask generated earlier

#Parametric Magic
std_patch_R = np.std(patch_R.flatten())
mean_patch_R = np.mean(patch_R.flatten())
std_patch_G = np.std(patch_G.flatten())
mean_patch_G = np.mean(patch_G.flatten())
#Making the gaussian work
def gaussian(p,mean,std):
    return np.exp(-(p-mean)**2/(2*std**2))*(1/(std*((2*np.pi)**0.5)))

Now let’s check the distribution of the patch that we have made using our codes above

x = np.linspace(0,1)
y = gaussian(x,mean_patch_R,std_patch_R)
plt.plot(x,y)

Let’s test our data out in the RG Chromaticity space

#Visualization of R space and G space
#R space
r_test = np.tile(np.linspace(0,1,64),(64,1))
plt.imshow(r_test)
#G space
g_test = r_test.transpose()
plt.imshow(g_test)

#Test sample R and G space into the gaussian distribution with the generated patch
#R space with patch
test_R = gaussian(r_test,mean_patch_R,std_patch_R)
plt.title('R Space and patch')
plt.imshow(test_R)
#G space with patch
test_G = gaussian(g_test,mean_patch_G,std_patch_G)
plt.title('G Space and patch')
plt.imshow(test_G)

prob_test=test_R * test_G
plt.imshow(prob_test)

The image above shows that the color of the patch has a probability of being part of the painting by using only the R coordinate. Let’s try masking our painting using only the R Chromaticity space

prob_R = gaussian(paint_R,mean_patch_R,std_patch_R)
plt.imshow(prob_R)

Now let’s try it out using the G Chromaticity space

prob_G = gaussian(paint_G,mean_patch_G,std_patch_G)
plt.imshow(prob_G)

Multiplying the R and G space gives us the following output

prob=prob_R * prob_G
plt.imshow(prob)

The combined images now indicate that there is an actual probability of the taken color being part of the reference patch that we have taken earlier. We can now segment the image properly by generating a new mask with a given set of probability values

prob_list = [0.2, 0.4, 0.6, 0.8]
fig, ax = plt.subplots(1,4, figsize=(10,10))
for i in range(4):
    ax[i].imshow(prob > prob_list[i], cmap='gray')
    ax[i].set_title(prob_list[i])
plt.show()

Integrating the mask at our original RGB picture isolates the body of water. For this example, we will use 0.8 as the threshold

mask = prob > 0.8
red = paint[:,:,0]*mask
green = paint[:,:,1]*mask
blue = paint[:,:,2]*mask
paint_masked = np.dstack((red,green,blue))
imshow(paint_masked)

2. Non-Parametric Segmentation

For the cases where the regions of interest are not easily approximated by our generated 2D Gaussian function, a non-parametric approach can be done instead. This works by using the 2D histogram from the painting and using histogram back-projection to mask the image with the computed histogram from the reference patch.

Now that’s been said, we can simply do all that with cv2! Here are some lines of code to accomplish just that. For this example though, we will only use a 1-D histogram, taking into account the Hue from the HSV color space

import cv2 as cv
#histogram and backprojection
def Hist_and_Backproj(val):

    bins = val
    histSize = max(bins, 2)
    ranges = [0, 180] # hue_range

    hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
    cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)

    backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)

    cv.imshow('BackProj', backproj)

    w = 400
    h = 400
    bin_w = int(round(w / histSize))
    histImg = np.zeros((h, w, 3), dtype=np.uint8)
    for i in range(bins):
        cv.rectangle(histImg, (i*bin_w, h), ( (i+1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255), cv.FILLED)
    cv.imshow('Histogram', histImg)
#getting image and transforming to hsv
src = cv.imread('medium/painting.jpg')
hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)
ch = (0, 0)
hue = np.empty(hsv.shape, hsv.dtype)
#using only 1-D histogram (Hue)
cv.mixChannels([hsv], [hue], ch)
#creating trackbar to change bin values
window_image = 'Source image'
cv.namedWindow(window_image)
bins = 25
cv.createTrackbar('* Hue  bins: ', window_image, bins, 180, Hist_and_Backproj )
Hist_and_Backproj(bins)
#show image and allow user to close the program
cv.imshow(window_image, src)
cv.waitKey()


Image Differencing

Now this is a fun subject to talk about, especially with kids. Image Differencing is when instead of looking at color spaces, we want to determine changes or movements in our videos or images, pretty much like how kids play "spot the difference" when shown 2 almost identical pictures.

image1 = imread('medium/19a.jpg')
image2 = imread('medium/19b.jpg')
fig, ax = plt.subplots(1,2,dpi=100)
ax[0].imshow(image1)
ax[1].imshow(image2)

One way to go about image differencing is to simply transform both images into grayscale, and literally subtract both images from one another. The resulting output gives out the changes between the two images

#grayscale
from skimage.color import rgb2gray
image1_gray = rgb2gray(image1)
image2_gray = rgb2gray(image2)
fig, ax = plt.subplots(1,2,dpi=100)
ax[0].imshow(image1_gray,cmap='gray')
ax[1].imshow(image2_gray,cmap='gray')

Now let’s spot the difference

diff = image1_gray - image2_gray
imshow(diff)

And there you have it! Chromaticity Segmentation and Image Differencing


Interested in my work? You can see more stories over at my profile

Nico Aguila – Medium


Related Articles