Hands-on Tutorials, A comprehensive guide to

Detecting Objects in Flutter

Using tflite and SSD-MobileNet

Photo by Weston MacKinnon on Unsplash | Just some objects :D

Introduction

In this guide, we will be developing an application in Flutter using the tflite package and a pre-trained SSD-MobileNet model, capable of detecting objects in images and real-time camera stream. This application is capable of detecting objects offline. We will also be able to take pictures from within the app and feed it to the model for detection.

rupakkarki27/object_detection

Please note that I am no longer maintaining this project because of my busy schedule. Breaking changes introduced in the recent version of Flutter and dependencies may break the app. But you’re free to fork the repo and change things your way. Thanks!

Setting up the project

Setup Flutter on your machine if you haven’t already. This guide in the Flutter website is a great place to start. After setting up Flutter, create a new project by typing the following in your terminal.

$ flutter create object_detection
$ cd object_detection

You can now open your project in your favorite editor.

Installing packages

In your pubspec.yaml file, under the dependencies section add the following to install the required packages.

image_picker: ^0.6.7
tflite: ^1.1.1
camera: ^0.5.8+7

We use the image_picker for picking images from gallery, tflite for running our model, and the camera package for controlling the camera. You can read more about them on pub.dev.

In your app level build.gradle file (/android/app), under the android block, add the following lines to make sure Flutter does not compress your models or the tflite package.

aaptOptions {
  noCompress 'tflite'
  noCompress 'lite'
}

Models

You can download the SSD-MobileNet model and its labels from the GitHub repo mentioned above or from anywhere and place them in your project. Then, in your pubspec.yaml file add your assets.

assets:
  - assets/models/ssd_mobilenet.tflite
  - assets/models/labels.txt

Initializing Cameras

The main function of this app is created in such a way that all the cameras are initialized right after the application starts. Then, the camera descriptions are passed on to all the classes that require it.

Main Function that also initializes the cameras

Object Detection in Images

To detect objects in images, we first need to load the model using the Tflite.loadModel method available in the tflite package. Then, we need to get the images from either our gallery or take one from within the app by launching the camera. The camera package provides the getImage method that can be used to do both.

After the image is loaded, we feed it into our model using the Tflite.detectObjectOnImage method. This method returns the detected class, confidence and points of interests that will help us draw bounding boxes around the objects.

Functions for various operations

Drawing bounding boxes

The basic idea of drawing bounding boxes around the detected object is to use the POIs from the model and the image dimensions.

Using Stack widget, we can place the bounding boxes on top of the image. Inside a Positioned widget, we place a Container that has borders all around it. We can also display the detected class and its accuracy as percentage by simply adding a Text widget and converting the confidence to percentage.

Code for the bounding boxes

The Static Image detection page has two Floating Action Buttons. Using one, we can pick images directly from the gallery and run our model on it. Using another, a camera is opened and we can take picture and feed it to our model.

Model detecting objects in images (Image by Author)

Real Time Object Detection

Detecting objects in real time is somewhat similar to detecting objects in images. Each frame in the video stream is treated as an image and the model is run on it. The model spits out results per frame and the bounding boxes are redrawn over and over on top of the image stream.

The camera controller object in the camera plugin has a method called startImageStream that can be used to feed frames to our model. Inside this method we can call the Tflite.detectObjectOnFrame method that takes the camera stream and runs it through our model. The rest of the method is the same i.e. the model returns detectedClass, confidence and the POIs and we use them to draw bounding boxes on top of the camera stream.

Real time detection

Real Time detection (GIF by Author)

Conclusion

We developed an application capable of detecting objects using Flutter, the tflite package and a pre-trained model. The application is not perfect and may not work on all the images. We can improve this by training our own model that can detect more classes with increased accuracy or use some other powerful model.

Despite this hurdle, the application performs really good at images and videos lit well. This is a really good project for someone who is looking to combine their knowledge of Machine Learning and Flutter. We can also use Google’s Teachable Machine to train models to classify or detect specific types of images.

The tflite plugin currently only supports SSD-MobileNet and YOLO for object detection.

You can check out the project repo here. If there are any issues, feel free to open issues or create a PR. Any feedback or suggestions are welcome and encouraged.

You can also look at my portfolio here.