Tutorial: Selfie Filters Using Deep Learning And OpenCV (Facial Landmarks Detection)

This is a tutorial on how to build a python application that can put various sunglasses on a detected face (I am calling them ‘Selfie Filters’) by finding the Facial Keypoints (15 unique points). These keypoints mark important areas of the face — the eyes, corners of the mouth, the nose, etc.

Project Description

OpenCV is often used in practice with other machine learning and deep learning libraries to produce interesting results. Employing Convolutional Neural Networks (CNN) in Keras along with OpenCV — I built a couple of selfie filters (very boring ones).

Facial key points can be used in a variety of machine learning applications from face and emotion recognition to commercial applications like the image filters popularized by Snapchat.

You can access the full project code here:

acl21/Selfie_Filters_OpenCV

Working Example

Code Requirements

The code is in Python version 3.6, uses OpenCV and Keras libraries.

Follow this medium post to install OpenCV and Keras in Python 3.

Data Description

This dataset on Kaggle allows us to train a model to detect the facial keypoints given an image with a face.

It was provided by Dr. Yoshua Bengio of the University of Montreal.

Each datapoint in the dataset contains space-separated pixel values of the images in sequential order and the last 30 values of the datapoint represent 15 pairs of coordinates of the key points on the faces.

So we just have to train a CNN model to solve a classic deep learning regression problem. Check out the loss function list for regression problems here.

Code Explanation

Step 1: Train A Convolutional Neural Network Model

1.1 Define Model

We first define the model architecture in a separate python file for convenience — my_CNN_model.py

We write a method to create a network architecture.

1.2 Compile, Save, Fit And Load

We now write methods to compile, save, fit and load the models

1.3 Model Builder (Wrapper)

We now write a model builder that loads the data and calls the above methods.

Step 2: Initialize Stuff

We now create a new python file — shades.py in which we write code to read the webcam input, detect faces, use the CNN model we built, etc.

Step 3: Detect Faces In The Input

Once you start reading the input from the webcam, detect faces in the input using the Cascade Classifier object we initialized earlier.

Step 4: Create Filter Switch Trigger

Here, I used a blue bottle cap as a filter switch. To detect the blue cap we write code to find the blue contours in the image. We pass the blueLower and blueUpper HSV ranges we defined in Step 2.

Once we create the blueMask, that is supposed to find blue things in a video, we use OpenCV’s cv2.findContours() method to find the contours.

The above code checks if there are any contours (blue bottle cap) in the frame, once found it checks if the center of the contour in touching the ‘Next Filter’ button we created in this step. If touched, we change the filter (increment filterIndex by 1)

Step 5: Detect Facial Keypoints Using The Model

We use the face cascade we initialized earlier to locate faces in the frame. We then loop over each and every face to predict the facial keypoints.

Before we pass the detected face to the model, it is necessary that we normalize the input because normalized images were what our model was trained on, resize it to 96 x 96 images because that is what our model expects. The above code does the same.

Step 6: Put Shades On The Faces Using The Facial Keypoints

Once we detect facial key points, we can use them to do all sorts of cool things. For example, you can use the nose keypoint to add a mustache, lips key points to maybe add color to them, etc.

Here, we used 4 of the key points to configure the width and height of the shades on the faces. And, face_resized_color has shades and maps back to frame and face_resized_color2 has keypoints and maps back to frame2. We then display both of them.

In the end, we just clean stuff up.

You can access the full project code here:

acl21/Selfie_Filters_OpenCV

Execution

1. Download The Data

Download the data from here and put it in a folder named ‘data’ in your project directory. Make sure the ‘data’ folder contains — training.csv and test.csv

2. Build The CNN Model

> python model_builder.py

This calls my_CNN_model.py so make sure you have that file in your project directory.

4. Run The Engine File

> python shades.py

5. Grab A Blue Bottle Cap

And have fun!

Conclusion

In this tutorial, we built a deep convolutional neural network model, trained on the facial keypoints data. We then used the model to predict facial key points for the faces detected in the input webcam data. We also created a switch using contours, which allows us to iterate through other filters with gestures. I encourage you to tweak the model architecture and see for yourself how it affects keypoints detection.

This data only had 15 key points, there are several other datasets out there that have over 30 keypoints labeled on the face.

Awesome right?

Applications of Facial Keypoint Detection:

• Facial feature detection improves face recognition
• Male/Female Distinction
• Facial Expression Distinction
• Head pose estimation
• Face Morphing
• Virtual Makeover
• Face Replacement

Hope this tutorial was fun. Thanks for reading it.

Live and let live!
A