The world’s leading publication for data science, AI, and ML professionals.

How to make your own Instagram filter with facial recognition from scratch using python

Over the past 10 years, Facial recognition technology has developed rapidly and has quickly developed a variety of uses. From the…

Hands-on Tutorials

Image by teguhjati pras; Creative Commons License via Pixabay
Image by teguhjati pras; Creative Commons License via Pixabay

Over the past 10 years, Facial recognition technology has developed rapidly and has quickly developed a variety of uses. From the utilitarian (unlocking your phone), to playful (Instagram filters), to the controversial (security, surveillance, and policing), our faces can be used by tech in many ways. Curious about how to create some facial recognition tech yourself? Facebook offers the SparkAR platform to create facial recognition filters for Facebook and Instagram. However, we can pretty easily create one ourselves using the OpenCV package in python, so we can use facial recognition anywhere.

Background Info: How do computers "see"?

To a computer, images are just a series of numbers indicating where pixels are located. Computer programmers can create algorithms that teach computers how to identify unique features in images. Interestingly, computers "see" similarly to how humans solve jigsaw puzzles. Consider the image below:

Image by Bill Selak; Creative Commons License via Flickr
Image by Bill Selak; Creative Commons License via Flickr

Puzzle pieces that are apart of the clover are easy to place because they have distinct shapes and can only logically go in one place. Edge pieces also have a distinct flat feature limiting the possible places it could go to only a set of possibilities along the edge. The remaining pieces could really be anywhere. This process is called feature image extraction, where the user identifies features of the image to help them construct the overall image. Computers can imitate this line of thinking through trends in the pixel data. Data scientists use machine learning algorithms to teach computers how to identify these and increasingly complex trends. For facial recognition, the eyes are identified first (one of the most distinctive features found on a face), and then the rest of the features found on a face are mapped out using templates.

Step 1: Find Faces in a Picture

Now that we know the basics of how Computer Vision works, we can begin to build our filter. First, let’s find faces and eyes in a static picture. Begin by installing on your computer and then importing OpenCV (an open-source python package for image processing) into a py file. All image structures in OpenCV are can be converted to and from NumPy arrays so it may be useful to import NumPy as well. Once you’ve installed OpenCV, you should have access to .xml files that contain facial recognition and other image processing algorithms. For this tutorial, we’ll use an algorithm called the Haar Cascade for faces and eyes. If you are having trouble finding the directory where these .xml files are, I suggest a quick file search for "haarcascade". Once you have found the path to the directory where your Haar Cascades are stored, call CascadeClassifier and pass the path through it:

Great, now that we’re set up, we can load in the images and look for faces. Note that Haar Cascades and many Facial Recognition algorithms require images to be in grayscale. So, after loading in the image, convert it to grayscale, and then use the face_cascade to detect faces. After you’ve got the faces, draw a rectangle around it and search within the facial region for eyes. Then draw rectangles around each eye.

Passing a stock image from Unsplash through the program outputs:

Photo by Gustavo Alves on Unsplash; altered by author
Photo by Gustavo Alves on Unsplash; altered by author

Cool! That worked well! Note that sometimes the Haar Cascade does not properly recognize faces and eyes, particularly if someone is not fully facing the camera, and sometimes it will think that corners of mouths and noses are eyes. If this happens, for this tutorial I suggest using another photo. Otherwise, you’ll need to create a whole other algorithm yourself instead of using OpenCV’s Haar Cascade.

Step 2: Create your image filter

For my filter example, I’m going to use witch hats because Halloween is this weekend. Editing the code above, we can read in the image a witch hat along with our stock photo. Because we are placing the hats on top of the stock photo, we’ll have to get the coordinates of where we want to put the hats. We’ll also have to clear out which parts of the witch image we actually want to use. We can do this by using the cv2.threshold() and cv2.bitwise_not() methods to identify which part of the witch image we want to keep and which we do not (this is called a mask).

Then, for each face, we have to place where the witch’s hat will go. Identify the coordinates for the boundaries of the face region using the height and width of the face. Then resize the witch image (or whatever filter you chose) to fit on the face region and select appropriate coordinates to place it on. You may have to fiddle around with these coordinates to get the image you chose to land it the right spot. In my witch example, I had to move up the filter/witch image, as seen below in witch_y1, because the brim of the hat should land on the person’s forehead. If I had not done that, the hat image would have been placed exactly where the face region was. Be sure to check to see if your filter image goes out of the frame of your main image. Lastly, use the masks to carve out the place to put your filter.

Running this new script on our stock image:

Photo by Gustavo Alves on Unsplash; altered by author
Photo by Gustavo Alves on Unsplash; altered by author

We’ve successfully built the facial recognition filter for static images!

Step 3: Applying to live video feed

So far we’ve identified faces in a static photo and placed a filter image on top of them. Lastly, we need to do this with live moving images. The code is the same as before, but this time instead of a static image, we let OpenCV access to our computer’s camera. We’ll also need to wrap the faces loop we used in our previous two steps in a while loop to keep it finding faces in every frame of live video we feed it. The user can quit out of the program by pressing the letter ‘q’.

Step 4: Screen Capture or Screen Record

At this point, the easiest thing to do is just use our computer’s built-in screen capture or screen record to save the filtered video feed from our camera.

GIF created by Mitchell Krieger
GIF created by Mitchell Krieger

Alternatively, if you want to do everything with code or have more control over the capture or recording, you can use PIL or PyAutoGUI to capture your screen through code.

Curious about my original scripts? Check out this GitHub Repository


Related Articles