The world’s leading publication for data science, AI, and ML professionals.

Teaching New Tricks to an Old Camera, Part II

Faster with TF Lite, MobileNet SSD v2, Coral Edge TPU and OpenCV reading thread

(image by author)
(image by author)

Early in August I wrote about my "driveway camera" project. Soon after I got a Coral Edge TPU to tinker with, and while really intended for other projects, I also had to see how it did with that :). Here is a quick update:

To use this TPU, you need to first convert your model to TensorFlow Lite format and quantize it. I tried converting my YOLOv3 model, but quickly ran into an issue that caused the converter to dump core. While this was being worked on, I thought I’d try out the pre-trained MobileNet SSD v2 Coral makes available; there’s one version that was trained on the same (COCO) dataset, so at least the detected classes are the same.

Wanting my script to be able to do it all so that I could compare, I refactored it, extracting classes to accommodate the variations:

  • TfEngine and TfliteEngine are instantiated with a model handle the details of pushing data in and getting data out. For TensorFlow it mostly invokes model.predict(), but TF Lite takes a bit more work;
  • YoloDetector and MobilenetSsdDetector implement a detect() method that pre-processes the input image, sends it through the engine, and post-processes the output into a list of detection boxes (the neural networks are sufficiently different on output to make these classes worthwhile). I also have a DummyDetector that does nothing, to easily "baseline" the non-detection overhead;

This way, I can instantiateYoloDetector(TfEngine(model),...), or MobilenetSsdDetector(TfliteEngine(model),...), etc., and pass that to the code that processes a video file. Yup, nothing to see here, just some less-ubiquitous-than-you’d-think 1950s object oriented technology 🙂

Digital video encoding is pretty fancy and OpenCV needs a bit of CPU time to turn that back into a sequence of frames, so I coded a version of my VideoReader class that reads in its own thread, pushing into Python‘s conveniently synchronized[queue.SimpleQueu](https://docs.python.org/3/library/queue.html)e. The new class is named, you guessed it, ThreadedVideoReader.

The numbers

Without further ado, here are some of the times I measured. I added a YOLO 320×320 configuration to make it more comparable in input volume to MobileNet SSD. Note that my code still skips over frames as I described in the original article:

Wall clock times to process a "representative" 1-minute video clip.
Wall clock times to process a "representative" 1-minute video clip.

Cable matters

One pitfall I fell into as I tried out the Tpu was not to use a good enough USB cable. I was originally using a short USB-C cable that I think came as a charging cable for my iPad, and kept seeing subpar performance. It turns out, with that cable both my mac and my Linux PC saw the TPU as an USB 2.0 device, which makes a big difference for the data transfer. With a better cable, it is recognized as a USB 3.1 device (Google has vendor ID 18D1):

$ usb-devices
[...]
T:  Bus=06 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#=  3 Spd=5000 MxCh= 0
D:  Ver= 3.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 9 #Cfgs=  1
P:  Vendor=18d1 ProdID=9302 Rev=01.00
C:  #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=896mA
I:  If#=0x0 Alt= 0 #EPs= 6 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)

Daemon-izing it

If you feel systemd tries to do too much and is too complicated, I’m with you. Still, getting this running as a service wasn’t so bad. First you define the service, say, in /etc/systemd/system/camera.service:

[Unit]
Description=Camera object detection
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=juan
WorkingDirectory=/mnt/camera/detect
ExecStart=/usr/bin/env python /home/juan/camera/camera.py --mark
[Install]
WantedBy=multi-user.target

and then run systemctl enable camera, and reboot (to test). Or start it manually with systemctl start camera.

The Code

Here‘s the latest and greatest, for now. Enjoy! 🙂


Related Articles