The world’s leading publication for data science, AI, and ML professionals.

Streaming to Apple Home with a custom AI layer

How to use an old webcam as a modern security camera

Photo by Mark de Jong on Unsplash
Photo by Mark de Jong on Unsplash

Introduction

Home automation systems are becoming nowadays more and more popular. One thing one can implement in a smart home is a security camera.

I choose not to spend money on a brand-new camera, but to use a simple webcam and stream it to the remote control app.

The interesting stuff is that with this custom solution is that it would be easy to add layers of information on the images and as far as we are data scientist the limit is the sky (or the camera resolution).

Setup

Here I will describe a simple setup with a webcam and a RaspberryPi connected to a Neural Compute Stick.

They can be assembled together to build a web-server with incorporated AI. The hardware connections are trivial, one has only to take care to attach the NCS to the Raspberry’s USB3.0 port to obtain better performances.

The webcam register the video, the Pi gets the frames and, using the Compute Stick, add some information above them.

Setup. Image by Author.
Setup. Image by Author.

Stream the frames: a webserver

We need to send the frames and to do so we will use a simple webserver serving ffmpeg compatible frames to a given port via http.

The core of the webserver is in the following code: each time the enerator is called a frame is captured, a model called and the resulting image is served, and it is all very fast.

def generator():    
    import io
    global model
    cap = cv2.VideoCapture(0)
    while True:
        ret, frame = cap.read()
        frame = model.process(frame)
        encode_return_code, image_buffer = cv2.imencode('.jpg', frame)
        io_buf = io.BytesIO(image_buffer)        yield (b'--framern'+               b'Content-Type: image/jpegrnrn' + io_buf.read() + b'rn')
@app.route("/video")
def video():
    return Response(generator(), mimetype='multipart/x-mixed-replace; boundary=frame')

Using Flask and OpenCV it is possible to get sequences of video frames from a camera connected via USB to the RPi and then stream them to a webpage accessible with many different tools.

AI layer

When the frame is captured it is possible to parse it using an image classification model and add to the frames information on which object is present in the image.

Using openVINO the classification process would be as easy as

exec_net.infer(inputs={self.input_blob: images})

Another more interesting possibility is to use a YOLO model. Classical approaches apply the model to an image at multiple locations and scales. High scoring regions of the image are considered detections.

YOLO uses a totally different approach. It applies a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. And so it is possible to identify multiple objects in an image very efficently.

Here I described two examples: a classifier and YOLO, but, of course one can also run any model simply converting it to a format compatible with the Neural Compute Stick.

Send video to mobile devices: Homebridge

Ones one has the video frames it is possible to stream them to other devices, my favourite approach is to stream it to apps devoted to home management, so they can be integrated with others devices eventually. In this case I used Homebridge.

Homebridge is a very powerful tool to connect every kind of devices, even custom ones as we will see soon to Apple Home infrastructure.

Homebridge

ffmpeg plugin

In this example, I choose a plugin that uses ffmpeg to read those frames.

Sunoo/homebridge-camera-ffmpeg

This should be configured to read frames from localhost on, for instance, port :5000 (or the one configured in Flask). It is also possible to choose a name for the device and other settings.

"accessories": [
{
            "name": "Camera FFmpeg",
            "cameras": [
                {
                    "name": "raspiwebcam",
                    "videoConfig": {
                        "source": "-f mjpeg -i http://localhost:5000/video"
                    }
                }
            ],
            "platform": "Camera-ffmpeg"
        }
    ]

Apple Home app

When Homebridge is set up, the server is up and running we are ready to inspect the images from the webcam in the Apple Home app.

As you can see in the following screenshot (the image from the webcam are low quality, I know)

An image classifier based on Inception can add information on which object is being currently seen by the model.

Apple Home screenshot. Image by Author.
Apple Home screenshot. Image by Author.

Running yolo

YOLO: Real-Time Object Detection

A simple script can be used to process frames and detect objects on frames

yolo on Apple Home. Image by Author.
yolo on Apple Home. Image by Author.

Now we are ready to catch every interesting and cute stuff on the camera!

Code

https://github.com/fvalle1/ai_server/tree/raspbian is a repository with all the scripts and the pieces of code need to let all of this work properly.


Related Articles