Image Processing with Python — Template Matching with Scikit-Image
How to identify similar objects in your image
Template matching is a useful technique for identifying objects of interest in a picture. Unlike similar methods of object identification such as image masking and blob detection. Template matching is helpful as it allows us to identify more complex figures. This article will discuss exactly how to do this in Python.
Let’s get started!
As always, begin by importing the required Python libraries.
import numpy as np
from skimage.io import imread, imshow
import matplotlib.pyplot as plt
from matplotlib.patches import Circle, Rectangle
from skimage import transform
from skimage.color import rgb2gray
from skimage.feature import match_template
from skimage.feature import peak_local_maxGreat, now let us load the image we will be working with.
leuven = imread('leuven_picture.PNG')
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(leuven);
The image above is of the Leuven Town Hall I took some years ago. Its highly decorative window arches are definitely a sight to behold. For our task let us try to use template matching to identify as many of them as possible. Our first step of course is to convert the image to grayscale.
leuven_gray = rgb2gray(leuven)
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(leuven_gray);
Excellent, now let us pick out one of the windows and use it as a template. To do this we simply have to cut out that slice of the image.
template = leuven_gray[310:390,240:270]
imshow(template);
At this point we can feed the template into the match_template function of Skimage.
resulting_image = match_template(leuven_gray, template)
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(resulting_image, cmap='magma');
The above is the result of using the match_template function. Put very simply, the brighter the section of the image, the closer of a match it is to the template. Let us see which section of the image the function thinks is the closest match to the template.
x, y = np.unravel_index(np.argmax(resulting_image), resulting_image.shape)
template_width, template_height = template.shape
rect = plt.Rectangle((y, x), template_height, template_width,
color='r', fc='none')
plt.figure(num=None, figsize=(8, 6), dpi=80)
plt.gca().add_patch(rect)
imshow(leuven_gray);
We can see that the image was able to correctly identify the perfect match for the template (to validate you can check with the slicing coordinates we used). This will definitely be useful in any task that would require you to search for an exact match of an object within an image.
Let us now see if we can get the function to identify the other windows as being more or less similar to our template.
template_width, template_height = template.shape
plt.figure(num=None, figsize=(8, 6), dpi=80)
for x, y in peak_local_max(result, threshold_abs=0.5,
exclude_border = 20):
rect = plt.Rectangle((y, x), template_height, template_width,
color='r', fc='none')
plt.gca().add_patch(rect)
imshow(leuven_gray);
We see that though the function does accurately identify several other windows. It also erroneously identifies several other objects that are clearly not windows. Let us see if we can cut down on the amount of false positives.
One way we can can remedy this is by making use of use of the homography matrix. I’ve written an article previously on how to make use of the transform.warp function in Skimage, but generally it warps the image and make it seem as if the image had been taken from another angle.
points_of_interest =[[240, 130],
[525, 255],
[550, 545],
[250, 545]]
projection = [[180, 150],
[520, 150],
[520, 550],
[180, 550]]
color = 'red'
patches = []
fig, ax = plt.subplots(1,2, figsize=(15, 10), dpi = 80)
for coordinates in (points_of_interest + projection):
patch = Circle((coordinates[0],coordinates[1]), 10,
facecolor = color)
patches.append(patch)for p in patches[:4]:
ax[0].add_patch(p)
ax[0].imshow(leuven_gray);for p in patches[4:]:
ax[1].add_patch(p)
ax[1].imshow(np.ones((leuven_gray.shape[0], leuven_gray.shape[1])));
points_of_interest = np.array(points_of_interest)
projection = np.array(projection)
tform = transform.estimate_transform('projective', points_of_interest, projection)
tf_img_warp = transform.warp(leuven, tform.inverse)
plt.figure(num=None, figsize=(8, 6), dpi=80)
fig, ax = plt.subplots(1,2, figsize=(15, 10), dpi = 80)
ax[0].set_title(f'Original', fontsize = 15)
ax[0].imshow(leuven)
ax[0].set_axis_off();
ax[1].set_title(f'Transformed', fontsize = 15)
ax[1].imshow(tf_img_warp)
ax[1].set_axis_off();
We can see that the image now faces forward. As we have mitigated the effect the angle has on template matching, let us see if we get better results. As before, let us first convert the image into grayscale and then apply the transform function.
points_of_interest = np.array(points_of_interest)
projection = np.array(projection)
tform = transform.estimate_transform('projective', points_of_interest, projection)
tf_img_warp = transform.warp(leuven_gray, tform.inverse)
plt.figure(num=None, figsize=(8, 6), dpi=80)
fig, ax = plt.subplots(1,2, figsize=(15, 10), dpi = 80)
ax[0].set_title(f'Original', fontsize = 15)
ax[0].imshow(leuven_gray, cmap = 'gray')
ax[0].set_axis_off();
ax[1].set_title(f'Transformed', fontsize = 15)
ax[1].imshow(tf_img_warp, cmap = 'gray')
ax[1].set_axis_off();
result = match_template(tf_img_warp, template)
plt.figure(num=None, figsize=(8, 6), dpi=80)
imshow(result, cmap=’magma’);
Now let us apply the exact same codes as before and see if we get better results.
template_width, template_height = template.shape
plt.figure(num=None, figsize=(8, 6), dpi=80)
for x, y in peak_local_max(result, threshold_abs=0.5,
exclude_border = 10):
rect = plt.Rectangle((y, x), template_height, template_width,
color='r', fc='none')
plt.gca().add_patch(rect)
imshow(tf_img_warp);
We can see that the algorithm can still identify every window on the image, however it still has those pesky false positives. To alleviate this, let us apply a filter the template matches. Below are some codes to do our data wrangling, apologies if they are slightly abtruse.
template_width, template_height = template.shape
matched_list = []
for x, y in peak_local_max(result, threshold_abs=0.50, exclude_border = 10):
rect = plt.Rectangle((y, x), template_height, template_width)
coord = Rectangle.get_bbox(rect).get_points()
matched_list.append(coord)
matched_patches = [tf_img_warp[int(match[0][1]):int(match[1][1]),
int(match[0][0]):int(match[1][0])] for match in matched_list]difference = [abs(i.flatten() - template.flatten()) for i in matched_patches]
summed_diff = [array.sum() for array in difference]final_patches =list(zip(matched_list,summed_diff))
statistics.mean(summed_diff)
After running the above codes, we can now create the filtered list of template matches. Again apologies if the code may not be that easy to follow.
summed_diff = np.array(summed_diff)
filtered_list_mean =list(filter(lambda x: x[1] <=
summed_diff.mean(), final_patches))
filtered_list_median =list(filter(lambda x: x[1] <=
np.percentile(summed_diff, 50),
final_patches))
filtered_list_75 =list(filter(lambda x: x[1] <=
np.percentile(summed_diff, 75),
final_patches))The above code should filter the matches by the mean difference, the median difference, and the 75% percentile difference. Essentially it will only hold matches that have absolute differences below those thresholds. The final step is to plot these out and see if the results have improved.
fig, ax = plt.subplots(1,3, figsize=(17, 10), dpi = 80)
template_width, template_height = template.shapefor box in filtered_list_mean:
patch = Rectangle((box[0][0][0],box[0][0][1]), template_height,
template_width, edgecolor='b',
facecolor='none', linewidth = 3.0)
ax[0].add_patch(patch)
ax[0].imshow(tf_img_warp, cmap = 'gray');
ax[0].set_axis_off()for box in filtered_list_median:
patch = Rectangle((box[0][0][0],box[0][0][1]), template_height,
template_width, edgecolor='b',
facecolor='none', linewidth = 3.0)
ax[1].add_patch(patch)
ax[1].imshow(tf_img_warp, cmap = 'gray');
ax[1].set_axis_off()for box in filtered_list_75:
patch = Rectangle((box[0][0][0],box[0][0][1]), template_height,
template_width,
edgecolor='b', facecolor='none',
linewidth = 3.0)
ax[2].add_patch(patch)
ax[2].imshow(tf_img_warp, cmap = 'gray');
ax[2].set_axis_off()fig.tight_layout()
We can see that all of them do look much better than the original image. However, we notice that though Mean and Median have far less false positives they also have far less true positives. The 75 Perc filter however is able to retain almost all the true positives.
In Conclusion
Template matching can be a tricky thing if the template is a particularly complex image. We must remember that though we as humans may interpret the image as a simple window, the machine only sees a matrix. There then two ways we can tackle this issue. One is by ensuring that the template is unique enough that false positives will be rare, the other is developing a sophisticated filtering system that is able to accurately remove any false positives from the data. A topic like this deserves several articles and in the future we shall go over some best practices when it comes to template matching. For now I hope you were able to learn how to make use of template matching in your own projects and can now think ahead of how to deal with the inevitable issues.
