Deep Learning for Image Classification on Mobile Devices

Mobile Image Classification App Development using Expo, React-Native, TensorFlow.js, and MobileNet

React is a popular JavaScript framework for building Web user interfaces. Reactive Native inherits and extends the component framework (e.g., component, props, state, JSX, etc.) of React to support the development of native Android and iOS applications using pre-built native components such as View, Text, TouchableOpacity, etc. The native code is typically developed using Xcode or Android Studio.

Expo is a framework and platform built around React Native and mobile native platforms that allow us to develop, build, and deploy mobile applications on iOS, Android, and web apps from the same JavaScript/TypeScript codebase. The JavaScript/TypeScript code can be developed using any text editor tool.

With the announcement of TensorFlow.js for React Native, it is now possible to train new machine learning and deep learning models and/or use pre-trained models for prediction and other machine learning purposes on mobile devices.

In this article, similarly to [3], I use Expo [4] with React Native [5] to develop a multi-page mobile application that uses TensorFlow.js [1] and a pre-trained convolutional neural network model MobileNet [2] for image classification on mobile devices.

This mobile application is developed on Mac as follows:

• using Expo to generate a multi-page application template
• installing libraries
• developing mobile application code in React JSX
• compiling and running

1. Using Expo to Generate Project Template

In order to use Expo CLI to generate a new project template automatically, first the Expo CLI needs to be installed:

npm install expo-cli

Then a new Expo project template can be generated as follows:

expo init react-native-deeplearning
cd react-native-deeplearning

The project name is react-native-deeplearning in this article.

As shown below, I choose the tabs template of Expo managed workflow to automatically generate several example screens and navigation tabs.

The TensorFlow logo image file tfjs.jpg [3] is used in this project and it needs to be stored in the generated ./asserts/images directory.

2. Installing Libraries

The following libraries need to be installed for the image classification app on mobile devices:

• @tensorflow/tfjs, that is, TensorFlow.js, an open-source hardware-accelerated JavaScript library for training and deploying machine learning models.
• @tensorflow/tfjs-react-native, a new platform integration and backend for TensorFlow.js on mobile devices.
• @react-native-community/async-storage, an asynchronous, unencrypted, persistent, key-value storage system for React Native.
• @tensorflow-models/mobilenet, pre-trained model that can take an image as input and returns an array of most likely predictions and their confidences.
• expo-gl, provides a View that acts as an OpenGL ES render target, useful for rendering 2D and 3D graphics.
• jpeg-js, a pure javascript JPEG encoder and decoder for node.js

npm install @react-native-community/async-storage @tensorflow/tfjs @tensorflow/tfjs-react-native expo-gl @tensorflow-models/mobilenet jpeg-js

In addition, the react-native-fs (a native filesystem access for react-native) is required by @tensorflow/tfjs-react-native/dist/bundle_resource_io.js:

npm install react-native-fs

The expo-camera (a React component that renders a preview for the device’s either front or back camera) is needed as well since it is used in @tensorflow/tfjs-react-native/dist/camera/camera_stream.js.

expo install expo-camera

3. Developing Mobile Application Code

As described before, as a start, first I used Expo CLI to generate example screens and navigation tabs automatically. Then I modified the generated screens and added a new screen for image classification. As a result (see next section for details), I created three screens:

• Introduction screen (see Figure 3)
• Image classification screen (see Figure 4)
• References screen (see Figure 5)

There are three corresponding tabs at the bottom of screen for navigation purpose.

This article focuses on the image classification screen class (see [7] for source code). The rest of this section discusses the implementation details.

3.1 Preparing TensorFlow, MobileNet Model, and Camera Access

The lifecycle method componentDidMount() is used to initialize TensorFlow.js, load the pre-trained MobileNet model, and get permission for accessing camera on mobile device after the user interface of the image classification screen is ready.

async componentDidMount() {
    await tf.ready(); // preparing TensorFlow
    this.setState({ isTfReady: true,});
    this.model = await mobilenet.load(); // preparing MobileNet model
    this.setState({ isModelReady: true });
    this.getPermissionAsync(); // get permission for accessing camera on mobile device
  }
getPermissionAsync = async () => {
    if (Constants.platform.ios) {
        const { status } = await Permissions.askAsync(Permissions.CAMERA_ROLL)
        if (status !== 'granted') {
            alert('Please grant camera roll permission for this project!')
        }
    }
}

3.2 Selecting Image

Once the TensorFlow library and MobileNet model are ready, the method selectImage() is called for choosing an image on mobile device for model prediction.

selectImage = async () => {
    try {
      let response = await ImagePicker.launchImageLibraryAsync({
        mediaTypes: ImagePicker.MediaTypeOptions.All,
        allowsEditing: true,
        aspect: [4, 3]
      })
      if (!response.cancelled) {
          const source = { uri: response.uri }
          this.setState({ image: source })
          this.classifyImage()
      }
    } catch (error) {
      console.log(error)
    }
}

3.3 Classifying Image

Similarly to [3], once an image has been chosen on mobile device, the classifyImage() method is called for image classification.

In this method, first the fetch API for TensorFlow React Native is used to load the selected image on mobile device. Then the method imageToTensor() is called to convert the loaded raw image data into a 3D image tensor. Finally the prepared MobileNet model is called to take the 3D image tensor as input and generates top 3 predictions with probabilities/confidences.

classifyImage = async () => {
    try {
      const imageAssetPath = Image.resolveAssetSource(this.state.image)
      const response = await fetch(imageAssetPath.uri, {}, { isBinary: true })
      const rawImageData = await response.arrayBuffer()
      const imageTensor = this.imageToTensor(rawImageData)
      const predictions = await this.model.classify(imageTensor)
      this.setState({ predictions: predictions })
    } catch (error) {
      console.log('Exception Error: ', error)
    }
}
imageToTensor(rawImageData) {
    const TO_UINT8ARRAY = true
    const { width, height, data } = jpeg.decode(rawImageData, TO_UINT8ARRAY)
    // Drop the alpha channel info for mobilenet
    const buffer = new Uint8Array(width * height * 3)
    let offset = 0 // offset into original data
    for (let i = 0; i < buffer.length; i += 3) {
      buffer[i] = data[offset]
      buffer[i + 1] = data[offset + 1]
      buffer[i + 2] = data[offset + 2]
      offset += 4
    }
    return tf.tensor3d(buffer, [height, width, 3])
}

Note that there are two versions of fetch API, one is React fetch API and the other is a fetch API for TensorFlow React Native.

It is important to be sure to import the correct fetch API for TensorFlow React Native as follows to avoid error like “filereader.readasarraybuffer is not implemented in react native”:

import { fetch } from ‘@tensorflow/tfjs-react-native’

3.4 Reporting Image Classification Results

Once the image classification is done, the method renderPrediction() is called to display the prediction results on the screen of mobile device.

renderPrediction = (prediction) => {
    return (
      <View style={styles.welcomeContainer}>
        <Text key={prediction.className} style={styles.text}>
          Prediction: {prediction.className} {', '} Probability: {prediction.probability}
        </Text>
      </View>
    )
}

4. Compiling and Running Mobile Application

The mobile application in this article consists of a react native application server and one or more mobile clients. A mobile client can be an iOS simulator, Android emulator, iOS devices (e.g., iPhone and iPad), Android devices, etc. I verified the mobile application server on Mac and mobile clients on both iPhone 6+ and iPad.

4.1 Starting React Native Application Server

The mobile app server needs to start before any mobile client begins to run. The following commands can be used to compile and run the react native application server:

npm install
npm start

If everything goes through smoothly, a Web interface as shown in Figure 1 should show up.

4.2 Starting Mobile Clients

Once the mobile app server is running, we can start mobile clients on mobile devices.

Since I use Expo [4] for development in this article, the corresponding Expo client/app is needed on mobile devices. The Expo app for iOS mobile devices is available in Apple Store.

Figure 2: Expo client app in Apple Store.

Once the Expo client app has been installed on an iOS device, then we can use the camera on the mobile device to scan the bar code of the react native application server (see Figure 1) to use the Expo client app to run the mobile application.

Figures 3, 4, 5 show the resulting screens of the mobile application on iOS devices (iPhone and iPad).

Figure 3: Introduction screen on iOS devices.

Figure 4: Image classification screen on iOS devices.

Figure 5: References screen on iOS devices.

5. Summary

In this article, I developed a multi-page mobile application for image classification on mobile devices using Expo [4], React JSX, React Native [5], TensorFlow.js [1], and a pre-trained convolutional neural network model MobileNet [2].

I verified the mobile application server on Mac and the mobile application clients on iOS mobile devices (both iPhone and iPad).

This mobile app can potentially be used as a template for the development of other machine learning and deep learning mobile apps.

The mobile application project files are available in Github [7].

References

1. TensorFlow.js for React Native
2. MobileNet
3. A. Mittal, Image Classification on React Native with TensorFlow.js and MobileNet
4. Expo
5. React Native
6. React
7. Y. Zhang, Mobile app project files in Github