Train on AWS Cloud / Test on Edge Hardware / Free & Zero Local Setup
A typical AI Workflow involves defining a concrete use case, preparing a dataset, training a model, integrating the inferencing component into an application, and selecting proper hardware to deploy in production.
While there are ample cloud services that simplify training, there is a gap in identifying the appropriate production deployment hardware along with a neural network model that yields the best possible application performance that fits both the budget and use case before making any actual investment.
In this article, we will walk through cloud training to edge inference application development phases with a bit of python code and visual examples that you can follow along and adapt for your use case.
We will go through the below end-to-end workflow:
- Train a Model: Leverage AWS Sagemaker (free-tier) to train a TensorFlow model – with the Rome Weather Image Classification dataset as an example acquired from Kaggle.
- Import Model & Test Data: Import the trained model and test data from the S3 bucket to preview and use with a simple python program that analyzes images on Intel® Developer Cloud for the Edge (free).
- Benchmark & Optimize Inference: Boost inference performance with the OpenVINO™ integration with TensorFlow (python) and compare performance against stock TensorFlow on multiple hardware.
…All within our browser and without any local setup. Along the way, we will briefly touch upon pointers to adapt the same flow for other use cases, e.g., other TensorFlow-instance segmentation or object-detection models.
1. Train a Model with AWS Sagemaker
For the image classification tasks, SageMaker requires our training data to be in the structure:
{PARENT directory}/{CLASS-NAME directories}/{IMAGES}
We will use a small (250 tiny images – 2MB) dataset to classify – cloudy, rainy, snowy, sunny, and foggy weather in Rome. The data explorer option on Kaggle is a great way to preview the dataset’s contents before downloading.
Dataset formats differ per task, e.g., object detection requires a JSON annotation file. Use the default dataset for each task as an example to prepare the dataset, or use the learn more links in the train section of the model in SageMaker Studio.
Download & extract the dataset locally. We will then move the classes inside a train directory, and cut-paste-rename 1 image per class in a test directory to quickly test inference on our trained model later.
Navigate to your S3 bucket and upload the parent Rome Weather directory to ensure the folder structure is preserved and usable for image classification training. Finally, navigate and select the Copy S3 URI for the train directory to enter while configuring training in the next step.
If you don’t have a SageMaker account, sign up for a free-tier account here and create an s3 bucket to work with.
Launch our SageMaker Studio instance, and navigate to the JumpStart menu to select a Fine-Tunable TensorFlow image classification **** model.
Use the Train section to point to our parent training dataset directory from the bucket and output directory. You can also further configure the hyper-parameters. For the free tier, select the ml.m5.xlarge instance in the deployment configuration for training.
For the next step, ensure your AWS IAM User is configured with programmatic access and AmazonS3FullAccess policy.
We will enter the S3 bucket name, Access Key ID, and a Secret Access Key for the next step. For more info, refer to the programmatic access & secret keys page.
2. Import S3 Data on Intel® Developer Cloud
Launch Intel® Developer Cloud for the Edge – JupyterLab instance by navigating to Overview > Get Started with JupyterLab.
If you don’t have an Intel Developer Cloud for the Edge account already, enroll for free here.
Use the menu on the left to open the Cloud Storage Connector option and enter your S3 bucket name, Access Key ID, and Secret Access Key to view your S3 bucket contents.
Multi-select files to import e.g. {trained-model-output-directory} > {output} > model.tar.gz and test images from each class of your trained model.
3. Benchmark & Optimize Inference:
The application integration step can involve writing code that loads a model, loads and prepares test images for the input layer, and extracts results from the output later. We will do just that – prototyping in a notebook cell in a pre-installed kernel supporting TensorFlow and verifying if the code runs on the development machine.
Additionally, we will measure and print out inference time excluding any image pre or post-processing time to focus on the hardware and model inference performance relationship.
git clone https://gist.github.com/945165defefa93360db1172472c55d93.git test
You can clone the notebook used in this article to test against any of the image-classification Sagemaker TensorFlow Jumpstart models. For other tasks, modify it accordingly for the resulting output e.g. bounding box for detection, or mask for segmentation.
We will then add a shell script, that activates the pre-installed virtual environment and invokes our python program but this time we will submit the job on any of the hardware hosted on Intel® Developer Cloud for the Edge. This can range from Intel® Core™ (i3, 5, i7, i9) to Xeon® spanning multiple generations of processors. Use the cell in the notebook, to identify the platform IDs to use while submitting our image classification job.
Our python inferencing code has extra lines that enable the same custom-trained Sagemaker Jumpstart TensorFlow model to run an accelerated inference with the OpenVINO integration as the backend.
import openvino_tensorflow as ovtf
ovtf.enable()
//Optional: default is CPU or GPU, VPU if the hardware supports it
//one line to run the same model on accelerator hardware.
ovtf.set_backend('<backend_name>')
Finally, submit our job to an edge node and run first with stock TensorFlow followed by another job with OpenVINO backend for TensorFlow to compare the inference performance gains with our test program.
Takeaways
With the above workflow example, we can continue experimenting with Hyperparameters to achieve ideal accuracy metrics on other image classification JumpStart models (e.g. MobileNetV2, EfficientNet…) in SageMaker Studio, quickly import to Intel Developer Cloud for the Edge, and evaluate throughput/latency on other Edge Nodes on Bare-Metal Workloads (across generations of Intel® Core™ i3, i5, i7 or Xeon® processors).
Additionally, as shown in the notebook with pre-installed OpenVINO™ integration with TensorFlow Jupyter kernel we can boost inference performance without altering the original model. The log output of submitted jobs in the JupyterLab Filesystem indicates the number of nodes of the model topology that are running with the OpenVINO backend to aid with the inference performance gains.
Above flow serves as a baseline for the remaining application integration steps.
Useful links:
-
Example Test/Train Kaggle Datasets: Rome Weather Image Classification (2MB), Donkey-Horses-Zebra(25MB), 26-ball-types(192MB)
- Intel Developer Cloud for the Edge
- Example Jupyter Notebook
- Amazon SageMaker and S3
- OpenVINO Integration with TensorFlow