iOS Computer Vision Object Detection with Turi Create

Computer Vision projects are everywhere.

Unfortunately, it isn’t always easy to train your computer vision models. Object detection in Keras requires some manipulation and writing the same code in Tensorflow or PyTorch requires some serious code jujitsu. Then even if you are able to train your model, good luck getting it onto a mobile device. Google Cloud’s AutoML solves some of this but easy iOS deployment can be elusive.

Apple has done an amazing job making computer vision for iOS more approachable with it’s Create ML tool. If you don’t care about hyperparameter tuning or if you are intimidated by Python or Keras then Apple provides a quick and easy way to just drag-and-drop your labeled images then start training. After training, the models can then be easily dropped into an iOS app to make it easy for others to use what you’ve created. As a starting point, Apple Developer documentation provides a great sample app for object detection.

But what if the model you’re trying to train is too resource intensive to build it on standard Mac hardware? What if you want to take advantage of a GPU to speed the training process? Based on the title of this post you can probably assume this is where Turi Create enters the fold.

apple/turicreate

To get a better understanding of what an end-to-end Turi Create project looks like, you can take a look at this repo for computer vision playing card detection, as well as this paper summarizing the project or this video overview.

mcgovey/compvision-playing-card-detection

For those new to Turi Create, it is a Python library that abstracts away the complexities of Tensorflow, PyTorch, or even Keras. Its Python API makes it easy to format data into the required pre-processing image format, then create a model that can run on different types of hardware (not only Macs like Create ML).

The best part about using something other than a laptop is that there is a free resource that can help with training using a GPU already available, Google Colab. Depending on the complexity of your model, you may need to upgrade your account to take advantage of high-RAM and faster GPUs with Google Colab Pro.

Project Walkthrough

Before you can do anything with Turi Create though, you have to manipulate your data into a format called an sFrame. What this means is that the images you plan to train your computer vision model on must have a format that outlines the image, the label, and the dimensions of the bounding box (if it’s an object detection model). This notebook walks through an example for an object detection model.

Once your data is in an sFrame, you’re ready to start building your model using Turi Create. This notebook walks you through those steps. As mentioned above, one of the ways you can speed up model training is by using GPUs. If you’re using Google Colab, you can do this by clicking Runtime > Change runtime type > then selecting the dropdown under Hardware Accelerator and changing it to GPU.

Once that setting has been changed, you still need to get Turi Create to recognize your GPU. On a linux system (which is what Google Colab is built on), you can do that with code block below (note there are other things you’ll want to do between these commands but I wanted to highlight the GPU-specific commands.

!pip install turicreate

!pip uninstall -y tensorflow

!pip install tensorflow-gpu

!export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

tc.config.set_num_gpus(-1)

To read more about the specifics of the implementation of the library check out the Turi Create documentation.

Once you have your data loaded into an sFrame and your environment setup, you can start training your model. You can decide how many iterations you want your model to run for. For your first run, you’ll want to start with a small number of iterations to make sure everything is working as expected.

# Create a model

model = tc.object_detector.create(train_data, max_iterations=10)

Training was that easy!

Now that you have your model trained, make sure you save it. If you want to come back to it later, then you’ll want to make sure you save it as a .model file. When you are ready to drop it in an iOS app, save it as a .mlmodel.

# Save the model for later use in Turi Create

model.save('mymodel.model')

# Export for use in Core ML

model.export_coreml('mymodel.mlmodel')

One last thing! You’ll want to see how well your model performed. You can use the model.evaluate(test_data) method to see how your model performed on your training data or out of sample data.

That wasn’t so hard! So next time you are trying to build a computer vision model and you don’t want to worry about training a model in Keras and worrying about converting formats, or Create ML is taking too long, think Turi Create!