Data Science in the Real World

Optimizing Face Detection on your browser with Tensorflow.js

A browser based face detector is just what you need

My robust browser based face detection results

Let’s jump on to the most important question.

Why this detector?

You might have come across a lot of face detection tutorials and projects. This one is different though. It is deployed on the browser and can be used anywhere from proctoring online exams to many different arcade games to using it for detecting face masks, blurring or improving the resolution of the face in real-time (since the region of interest is obtained) while also performing at quite high speeds. Machine learning on the web is the requirement of this age and shouldn’t be restricted to local systems. An alternative approach is face-api.js which uses the Multi-Task CNN but our detector has higher accuracy. Many other projects would be based on face detection models deployed on Flask Apps which are quite slow in comparison.

Tensorflow serving makes use of gRPC and Protobuf while Flask apps use REST and JSON. JSON relies on HTTP 1.1 while gRPC uses HTTP/2. HTTP 1.1 suffers from latency issues. A TCP handshake is needed for each individual request, and a large number of requests takes a significant toll on the time needed to load a page. HTTP 1.1 also suffers from Head of Line Blocking which restricts the number of connections to the same domain. With HTTP pipelining, you can send a request while waiting for the response to a previous request, effectively creating a queue. But that introduces other problems. If your request gets stuck behind a slow request then your response time will suffer. HTTP /2 maintains the basic premises and paradigms of HTTP but does away with optional parts of HTTP 1.1. REST supports only the request-response model available in HTTP 1.x. But gRPC takes full advantage of the capabilities of HTTP/2 and lets you stream information constantly. Protobuf is a binary format used to serialize data and is more efficient than JSON. Tensorflow serving can batch requests to the same model, which uses hardware (GPUs) more appropriately. Tensorflow serving has performance equal to code written in C/C++.

Moreover, Flask apps are written in Python whereas Tensorflow.js has Node.js or Chrome V8 engine as its server side node. In this project, we use Chrome’s V8 Javascript engine which is an open source high-performance JavaScript engine.

To get more intuition on the Chrome V8 engine, read this awesome blog at

Understanding How the Chrome V8 Engine Translates JavaScript into Machine Code

Tensorflow.js provides the right kind of hardware acceleration needed using the Web GL layer, which might be considered as a browser-side GPU. WebGL is a JavaScript API for rendering interactive 2D and 3D graphics within any compatible web browser without the use of plug-ins. WebGL is fully integrated with other web standards, allowing GPU-accelerated usage of physics and image processing and effects as part of the web page canvas.

Thus this client side serving face-detector proves to be quicker than the erstwhile flask face detection apps.

Tensorflow.js

Tensorflow.js is a machine learning library in Javascript. It is an open-source hardware-accelerated JavaScript library for training and deploying machine learning models.

It can be used to develop ML in the browser by using flexible and intuitive APIs to build models from scratch using the low-level JavaScript linear algebra library or the high-level layers API. It can also be used to develop ML in Node.js by running native TensorFlow with the same TensorFlow.js API under the Node.js runtime. Pretrained Tensorflow or Keras models can be used in the browser by the TensorFlow.js model converters. Tensorflow.js can also retrain pre-existing model using sensor data-connected to the browser.

To know more about Tensorflow.js , you can check out the official documenation.

TensorFlow.js | Machine Learning for Javascript Developers

To get pretrained models, you can clone the following official tfjs GitHub repository

tensorflow/tfjs-models

Prerequisites

• A text editor (optional) (Sublime, Brackets, Visual Studio Code, etc.)

You can download the Brackets editor from

A modern, open source code editor that understands web design

• Google Chrome

Google Chrome - Download the Fast, Secure Browser from Google

Implementation

To use the browser based face detector, check out my GitHub repository

sid0312/tfjs-face_detection

Steps for replication

If you want to get hands on and do things on your own, here’s a guide

• Create a starter HTML file

• Add the following lines to the html file tensorflow.js headers to import the tfjs model in the head tag

<script src=”https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script>
<script src=”https://cdn.jsdelivr.net/npm/@tensorflow-models/blazeface">
</script>

• Styling (optional): Create the div(s) for formatting and add the style-sheet link. Add the video element and the canvas
• Add the JavaScript file link in which the model is loaded
• Rename your file to index.html

Your index.html file should look like this

• Create a main.css file for some optional styling. Add the following code to it. Change as per convenience

You can get some lovely CSS at

CodePen

Note: The main.css load a GIF which is for intial loading and is included in the repository. Feel free to use your own GIFs.

• Create a blaze_pred.js file. Add the following code to it.

• Explanation

We create a self calling function in which we obtain the canvas and video elements by their id. To get the context for the canvas, we use the getContext() method.

To recieve the video feed using the navigator.getUserMedia method and adding the stream to the video source object.

video.play() is used to play the video but since we have set the visibility to false in the video element of the index.html file, the video feed cannot be physically viewed. The feed which is displayed is the canvas. The face predictions are drawn using the canvas context.

To synchronize the video feed and the canvas element, we add an event listener and call a function draw which draws the predictions to it.

• Create an asynchronous draw function with parameters as the video feed, context, the width and height of the canvas.

Draw the current frame of the video feed on it. Load the model using await blazeface.load(). Get the predictions on the model await model.estimateFaces() while setting the returnTensors parameter to false. Draw the bounding box on the canvas context using the predictions tensor. Obtain the confidence score using the predictions tensor and add the text to the canvas context relative to the bounding box. To call the draw function repeatedly set the timeout as 250 ms.

• Add the working files (index.html, main.css, blaze_pred.js) to a folder. Name the folder tfjs-face_detection. You can give any other name too.

Clone the repository

If you haven’t done the replication steps, type the following in your shell/terminal/command prompt

git clone https://github.com/sid0312/tfjs-face_detection

Using the detector

For Linux and Mac Users,

user@username:~$ cd tfjs-face_detection
user@username:~/tfjs-face_detection$ google-chrome index.html

For Windows Users,

C:\Users\username> cd tfjs-face_detection
C:\Users\username\tfjs-face_detection> index.html

Allow access to the webcam

Wait for a few seconds and you’re all set!

Conclusion

We have successfully detected our face on the browser using Tensorflow.js. You can create your own deep learning face model and convert it into a tfjs model using the tfjs model converter to increase its robustness.

Photo by Ian Stauffer on Unsplash

Author

Siddhant Baldota

Yeah! The one in the video feed is me! Check out my other articles:

OCT Image Segmentation using U-Nets

Counting people on your webcam using GluonCV and MxNet

Happy Computer Vision, Happy Deep Learning and Happy Deployment. Until next time!