After quite a long search, we have identified several products that can be used to ease the lift required to deploy deep learning models into production. Since our end application is not clear (though we have several ideas), the best approach to production-izing this model is to deploy it as a REST service that can be maintained separately from the data science lifecycle process that produces each model, and is simple enough for each data scientist to use. The ability to easily deploy REST endpoints is a key enabler for rapid iteration of models over time. Ubiops is a product that offers a hosting service for deep learning services, running on Kubernetes that greatly simplifies the process of creating and maintaining REST services.
![The output from the REST endpoint hosting the facial profile model. (Image by Author, original: Schniter, Eric, & Shields, Timothy. (2020). Participant Faces from a Repeated Prisoner's Dilemma [Data set]. Zenodo. http://doi.org/10.5281/zenodo.4321821)](https://towardsdatascience.com/wp-content/uploads/2021/03/1Zz1EhsUeiyOxdyDPV3OYwg.png)
Introduction
In this post, we use IPython Jupyter notebooks to deploy a REST endpoint with Ubiops. The model used in this demo is the face profile developed in our 2020–10–12 blog. That model is a facial profile meant as a tool to control for the effects of phenotypical attributes that can be observed from portrait-style images (pictures with faces). The REST endpoint will enable continuous development of the underlying model independently of the app or apps built on top of it. The majority of the bytes sent will be the weights from a trained Unet segmentation model which is 165MB, pickle-compressed. The last step in the demo shows how the endpoint can be used to source facial profiles by a developer customer.
Some alternatives to UbiOps that are commonly used are AWS Lambda or other similar serverless products, self-hosted "stateful" Flask apps, and other promising solutions marketed under the MLOps banner like Streamlit, Seldon, FastAPI and notably, Algorithmia that offers a "serverless GPU" product. Key concerns for model serving are: it needs to be low cost because we’re bootstrapping a product. It needs to provide a programmatic API and common-sense behavior around model publishing and consumption. Finally, it needs to work with Jupyter notebooks being the source
The code for this tutorial is found here: Github Repo
Deploy an Existing Model
The README has a more comprehensive description of the different artifacts used to deploy this endpoint. The steps were found on the Ubiops docs and in the Ubiops cookbook. This example differs slightly because one requirement was to keep the notebook as the system of record for the code. In the cookbook tutorial, the notebook code is populated from a python module and in this tutorial the python module is built from the notebook using the Fastai – nbdev project. The basic pattern is:

High-level Functionality
The implementation involves several steps. First, we create the Deployment
class. The implementation structure for the Deployment
class is defined by Ubiops and the best place to see that is in the cookbook. The Deployment
class becomes the entry point for the endpoint, all its dependencies can be accounted for in the deployment_package
. We zip up the deployment_package
, then register inputs and outputs of the deployment request using the Ubiops API. Finally, we will push your deployment_package
zip archive to Ubiops using the API. Below hits the high points, but the unabridged code is found in the Github Repo.
Deployment Class
import sys; import os
from pathlib import Path
import io
import json
import skimage
from skimage import color
import matplotlib.pyplot as plt
import numpy as np
from prcvd.img.core import (
TrainedSegmentationModel, MaskedImg
)
from prcvd.img.face import (
FacialProfile
)
from prcvd.core import json_to_dict
class Deployment:
def __init__(self, base_directory:str, context):
"""
Initialisation method for the deployment. It can for
example be used for loading modules that have to be kept in
memory or setting up connections. Load your external model
files (such as pickles or .h5 files) here.
:param str base_directory: absolute path to the directory
where the deployment.py file is located
:param dict context: a dictionary containing details of the
deployment that might be useful in your code.
It contains the following keys:
- deployment (str): name of the deployment
- version (str): name of the version
- input_type (str): deployment input type, either
'structured' or 'plain'
- output_type (str): deployment output type, either
'structured' or 'plain'
- language (str): programming language the deployment
is running
- environment_variables (str): the custom environment
variables configured for the deployment.
You can also access those as normal environment variables via os.environ
"""
print("Loading face segmentation model.")
self.basedir = Path(base_directory)
self.mod_fp = self.basedir/'model1.pkl'
self.output_classes = [
'Background/undefined', 'Lips', 'Eyes', 'Nose', 'Hair',
'Ears', 'Eyebrows', 'Teeth', 'General face', 'Facial hair',
'Specs/sunglasses'
]
with open(Path(self.basedir)/'ubiops_output_spec.json') as f:
self.output_spec = json.load(f)
self.output_mapping = {
obj['name']: obj['id']
for obj
in self.output_spec
}
self.size = 224
self.model = TrainedSegmentationModel(
mod_fp=self.mod_fp,
input_size=self.size,
output_classes=self.output_classes
)
Above, we load up the model into memory. Ubiops warns that this step happens after the endpoint is idle for a while, and the so-called cold-start can take some time. The base_directory
is the root level of the deployment_package
and you can see that we load several files that are shipped in that folder when Deployment
is instantiated. The next part is to define the Deployment.request
which calls model.predict
.
def request(self, data, attempt=1):
"""
Method for deployment requests, called separately for each individual request.
:param dict/str data: request input data. In case of deployments with structured data, a Python dictionary
with as keys the input fields as defined upon deployment creation via the platform. In case of a deployment
with plain input, it is a string.
- img: str, file path
- sampling_strategy: str, 'use_all' | ...
- align_face: bool, yes/no apply face alignment
- num_attempts: int, max attempts before failure (sometimes face alignment fails)
:return dict/str: request output. In case of deployments with structured output data, a Python dictionary
with as keys the output fields as defined upon deployment creation via the platform. In case of a deployment
with plain output, it is a string. In this example, a dictionary with the key: output.
"""
img = MaskedImg()
img.load_from_file(data['img'])
try:
profile = FacialProfile(
model=self.model,
img=img,
sampling_strategy=data['sampling_strategy'],
align_face=data['align_face']
)
except:
if not attempt > data['num_attempts']:
return self.request(
data=data,
attempt=attempt+1,
)
else:
return None, None
## .. make/save the plot .. (see github for full code)
row = profile.get_profile()
row['model_id'] = str(self.mod_fp) # for ubiops output type str
row = {**row, **{'img': str(outfp)}}
return {self.output_mapping[k]: v for k,v in row.items()}
model.predict
uses the user supplied input to calculate the facial profile and visualize it with the original image (see the repo). It’s possible to save files to the current working directory, and Ubiops takes care of cleaning them up. On the flip side, if you need to save the images submitted as inputs, you will need to add that code into the request.
Create deployment package
What is sent to Ubiops is a zipped archive that contains the Deployment
class and dependencies. If you have any private dependencies e.g. packages not on PiPy in private repos, you can add them into the deployment_package
manually. The part below was done in the notebook, but could be done from the command line as well.
Package up the Deployment class
The next part can be run from the notebook or from the command line. It converts the module generated by the notebook code (mod
) to a Ubiops private dependency. The necessary parts are the setup.py
and the mod
folder containing the python modules that are specific to this deployment. We remove the .git
directory and the notebooks, because we want to minimize the size of the deployment_package
.
cp -r .. ../../prod-mod-face-profile-cp
rm -rf ../../prod-mod-face-profile-cp/deployment_package/
rm -rf ../../prod-mod-face-profile-cp/*.zip
rm -rf ../deployment_package/libraries
mkdir ../deployment_package/libraries
mv ../../prod-mod-face-profile-cp ../deployment_package/libraries/prod-mod-face-profile
rm -rf ../deployment_package/libraries/prod-mod-face-profile/.git/
Get any other private dependencies
We need to grab any other private libraries to package up with this deployment. This library contains all the code that manipulates the trained Deep Learning model, and the code that calculates the face profile.
rm -rf ../deployment_package/libraries/prcvd
cd ../deployment_package/libraries && git clone @github.com/prcvd/prcvd.git">https://{user}:{pw}@github.com/prcvd/prcvd.git
cd ../deployment_package/libraries/prcvd && git checkout {branch}
rm -rf ../deployment_package/libraries/prcvd/.git/
Tip: delete the files not necessary to
pip install
the module.Warning: For this to work without many redundant files committed saved in the git repo, we don’t commit the
deployment_package
and other large artifacts such as*pkl
,*pth
and*zip
. See bottom of the.gitignore
for more details
Deploy
You can deploy the deployment_package.zip
using the Ubiops API. The first part is to define a "deployment" seen in the code below. The main part of this is to define the names and data types inputs and the outputs from the Deployment.request
. I chose to manage those in separate files since they would be used in several places in addition to defining the deployment here. This is the list of allowable Ubiops data types. I chose to make each of the items in my output a separate output object. Before connecting to the Ubiops Python API, it’s important to create a login and project.
Configure the Ubiops API Connection
This step requires you to manually obtain an api key with the Ubiops UI, and create a new project. Once you have the api key stored somewhere safe, you can connect.
client = ubiops.ApiClient(
ubiops.Configuration(
api_key={'Authorization': os.getenv('API_TOKEN')},
host='https://api.ubiops.com/v2.1')
)
api = ubiops.CoreApi(client)
deployment_template = ubiops.DeploymentCreate(
name='my_deployment',
description='description',
input_type='structured',
output_type='structured',
input_fields=[
ubiops.DeploymentInputFieldCreate(
name=str(obj['name']),
data_type=obj['data_type']['value'])
for obj in input_spec
],
output_fields=[
ubiops.DeploymentOutputFieldCreate(
name=str(obj['id']),
data_type=obj['data_type']['value'])
for obj in output_spec
],
labels={'demo': 'mod'}
)
Once the deployment exists, a deployment Version
is created, which is where the cluster details are defined (e.g. maximum instances, memory, idle_time, etc..), as follows:
version_template = ubiops.VersionCreate(
version='v1',
language='python3.6',
memory_allocation=3000,
minimum_instances=0,
maximum_instances=1,
maximum_idle_time=1800 # = 30 minutes
)
Finally, we can push the deployment_package.zip
to UbiOps with the following code:
file_upload_result = api.revisions_file_upload(
project_name='face-profile',
deployment_name='my_deployment',
version='v1',
file=zipfp
)
It’s worth viewing the notebook in the Github, which contains the full deployment code. In there, it shows how to utilize some of the exceptions generated by the Ubiops API to do thing like basic, but automatic upversioning.
Tip: The
deployment_name
andversion
identifiers can be controlled by the code in the notebook.
Checking the Deployment
One may use the API to check the status of a deployment, which requires repeatedly asking for status. Or, one may monitor status and logs in the Ubiops UI. The logs from the deployment can be seen below:

A Python Request Client
Once the model is deployed, it simplifies usage because now instead of needing to load up the model locally we can just call a REST endpoint. That pre-empts any issues with installation dependencies, which are in my experience the most frustrating types of problems preventing my models from being used by their intended audience. The call can be accomplished in one line as follows:
client = UbiOpsClient(
project_name='face-profile',
deployment_name='my_deployment',
deployment_version='v1',
input_spec='./ubiops_input_spec.json',
output_spec='./ubiops_output_spec.json',
api_key=os.getenv('API_TOKEN'),
api_host=os.getenv('HOST')
)
# >> out
{'0000': (52, 47, 49),
'0001': (133, 86, 112),
'0002': (179, 144, 159),
'0003': (33, 28, 34),
'0004': (106, 10, 34),
'0005': (118, 102, 117),
'0006': None,
'0007': (144, 122, 132),
'0008': None,
'0009': None,
'0010': (97, 87, 107),
'0011': (76, 72, 92),
'0012': (196, 163, 176),
'0013': (145, 122, 133),
'0014': (188, 157, 166),
'0015': 3.4990597042834075,
'0016': 6,
'0017': -0.0,
'0018': 1.596774193548387,
'0019': 99.0,
'0020': 62.0,
'0021': 18828,
'0022': 0.018748672190354792,
'0023': 0.04110898661567878,
'0024': 0.30359039728064585,
'0025': 0.0014871468026343743,
'0026': 0.013702995538559592,
'0027': 0.0,
'0028': 0.2953579774803484,
'0029': 0.0,
'0030': 0.0,
'0031': 0.006107924367962609,
'0032': 0.0053112385808370514,
'0033': 0.06161036753770979,
'0034': 0.09693010410027618,
'0035': 0.15604418950499258,
'0036': '/home/fortville/code/prod-mod-face-profile/deployment_package/model1.pkl',
'0037': '/home/fortville/code/prod-mod-face-profile/deployment_package/tmp.jpeg'}
Viewing Input/Output from the Model

Tip: In the past, using different technologies, I saved the "request client" code in its own module. The requester is often a data scientist with different permissions vs the engineer that creates the deployment.
Conclusions
Overall, UbiOps delivers on its promise of simplifying the deployment of a REST API service, and they provide a pretty generous free tier. UbiOps accepts a single Deployment class, and provides a simple directory structure to support its library dependencies. In this tutorial, a proof of concept for a continuous integration between a Jupyter Notebook and the UbiOps cloud is discussed. In that one can make changes, including to the model weights, test those changes and deploy – all in the jupyter notebook in a semi-automated fashion. The file where normally one would write the Deployment class is made standard by importing from the module created by the notebooks. Without that feature, one would need to maintain 1 version of the code in the notebook (e.g. "the prototype"), and another version in the python module (or just a python file) in production. Like the other productionization technologies I have used such as AWS Lambda, there is a learning curve where I had to partially backward engineer the tool and it required me to minorly rethink several aspects of my prototype design.
Future work includes integrating more with Github Actions such that the deployment_package.zip
creation and testing can be triggered by a new PR commit, or a merge into master branch. Actually, Ubiops has a recent article out describing how to integrate with github actions: here
All the code found in this tutorial can be found here