Hands-on Tutorials
My open-source GitHub script provides AI-based Filters which apply a rather new technology called Artistic Neural Style Transfer to the input stream of your physical webcam device. In contrast to traditional filters, these AI-based filters are feature-aware. Depending on what kind of features are apparent in the video, the AI adapts the output. In addition, these kinds of filters can be learned from any real-world image. Since the provided filters are directly applied on the webcam video stream, they can be used in all types of video conferencing tools, such as Zoom, Skype, Discord, MS-Teams….

In detail, my script sets up a virtual webcam device that applies Artistic Neural Style Transfer to the input stream of the physical webcam device. This new webcam device can then be used just like any physical webcam in any video-conferencing application. So far, this tool is only available for Linux, but I am working on a Windows version. To make the installation very convenient, all you need to do is build a Docker container. How this works in detail is explained in a simple step-by-step installation guide. Even your own custom styles can be easily created by training the _PyTorch fast neural style transfer example model_ to extract the style of a particular image.

What Neural Style Transfer is:
![Neural Style Transfer: the content features of the left image are combined with the style features of the center image to generate a style transferred image [by Theo Lagendijk, src, CC BY-NC-SA 2.0].](https://towardsdatascience.com/wp-content/uploads/2021/06/1qakwTUnI4K9p3rnxztR81g.jpeg)
This Deep Learning approach, originally developed by Leon Gatys et al. in 2016, can extract the style of a particular image and apply it to other images. The basic idea of this approach is to generate an image that, on the one hand, has similar content features as the content image and, on the other hand, tries to have the same amount of feature correlation as exists between the features of the style image. This feature correlation is a representation of the image’s style.
Technical hurdles and why this was not created earlier:
Since the invention of neural style transfer by _Leon Gatys et al. in 2016, no tool has been built to date that integrates this approach into a virtual Webcam device that can be used directly. One major hurdle has been that the processing time of good quality style transfer images was still too slow to achieve high frame rates in real-time. Another hurdle has been that it was difficult to access and customize webcam driver code to apply neural style transfer. Fortunately, two relatively recent technological achievements let me leap over these hurdles: First, Nvidia’s [TensorRT](https://developer.nvidia.com/tensorrt) libraries lead to a massive speedup in model inference time. I adapted the fast neural style transfer model from the PyTorch examples_ to be compatible with TensorRT, resulting in an inference time speedup of more than 75%. This makes it possible to run the webcam on HD at a frame rate of more than 25 frames per second on current Nvidia graphics cards. Secondly, the virtual webcam driver akvcam has paved the way for me to set up a virtual webcam on Linux that can be used like any physical webcam.
Have I now sparked your interest to try video conferencing with fancy, wacky, and individual styles? Get started right away by following the installation guide at https://github.com/MaximusMutschler/Virtual-Neural-Style-Transfer-Webcam-for-Linux. Requirements are Linux and a rather new Nvidia GPU.