How to Deploy Scikit-Learn Models to Azure Container Instances

1. Introduction

Now you have trained your scikit-learn models, what’s next? How can it be made available to downstream applications as an API? In this article we will examine how to train and deploy scikit-learn models as an API using MLFlow, Azure Machine Learning and Azure Container Instances. Here are brief description of the services that we will be using.

What is MLFlow?

MLFlow[1] is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLFlow offers 4 different components:

MLFlow Tracking: Record and query experiments: code, data, config, and results
MLFlow Projects: Package data science code in a format to reproduce runs on any platform
MLFlow Models: Deploy machine learning models in diverse serving environments
Model Registry: Store, annotate, discover, and manage models in a central repository

We will be using the MLFlow tracking feature to log parameters, results and artifacts from our machine learning experiments.

What is Azure Machine Learning?

Azure Machine Learning[2] is part of Microsoft’s Azure cloud computing platform which helps data scientist and engineers to manage their machine learning workflow.

What is Azure Container Instances?

Azure Container Instances (ACI)[3] is a managed service by Microsoft Azure which allows us to run containerized services that is load-balanced and has a HTTP endpoint with a REST API.

2. Setup Azure

Azure Account

We will be using Azure ML and ACI therefore an Azure account is mandatory. Sign up for a free Azure account and get $200 credits for the first 30 days if you are a new user.

Azure Machine Learning Workspace

The workspace[4] is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. The workspace keeps a history of all training runs, including logs, metrics, output, and a snapshot of your scripts. You use this information to determine which training run produces the best model.

A resource group is prerequisite for creating an Azure Machine Learning workspace.

1. Create a resource group

A resource group is a container that holds related resources for an Azure solution.

Go to https://portal.azure.com/
Find "Resource groups" under Azure Services or through the search bar

Create a new resource group

Fill in the details such as subscription, name of resource group and the region

2. Create Azure ML Workspace

Find "Machine Learning" under Azure Services or through the search bar.

Click on create

Fill in the blanks. Resource group is the one which we created in the previous step.

MLFlow tracking server is automatically created as part of the Azure ML workspace

3. Setup Training Environment

IDE

I am using Visual Studio Code, however you can use any IDE of your choice

Conda Environment

Ensure that miniconda3 installed on your machine. Create a python 3.7 conda environment from your command line interface. The environment name is arbitrary, I am naming it as general .

#command line 
conda create -n general python=3.7

Activate the conda environment. We will be doing all our development work in this environment.

#command line
conda activate general

Install the necessary packages

azureml-core==1.39
pandas==1.3.5
scikit-learn==0.23.2
cloudpickle==2.0.0
psutil==5.9.0
mlflow==1.24.0

Docker

Docker is require on your local machine as we will be deploying the webservice locally as a docker container for debugging before deploying it to Azure Container Instances.

Azure Machine Learning Workspace Configs

Download the Azure Machine Learning workspace configurations.

The config file is in JSON format and it contains the following information:

# config.json

{
    "subscription_id": "your-subscription-id",
    "resource_group": "your-resource-group-name",
    "workspace_name": "your-workspace-name"
}

We will need these information to connect to AML workspace for logging of experiments.

Project Structure

These are the notebooks and scripts in the project folder. We will walk through each of these in the next section.

train.ipynb: pre-processing, training and logging of experiments
register_model.ipynb: register model and environment to Azure ML
test_inference.ipynb: call the webservice (local or ACI) with sample data for testing purpose
local_deploy.ipynb: deploy the model locally using Docker
aci_deploy.ipynb: deploy the model to ACI
score.py: entry script to the model for inference
conda.yaml: contains dependencies for creating the inference environment

4. Project Walkthrough

This example takes us through the following steps:

Train a scikit-learn model locally
Track the scikit-learn experiment with MLFlow on Azure Machine Learning
Register the model on Azure Machine Learning
Deploy and test the model as a local webservice
Deploy and test the model as an ACI webservice

We will be using the Pima Indian Diabetes Dataset[5] from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. The datasets consists of several medical predictor variables and one binary target variable, Outcome. Predictor variables includes the number of pregnancies the patient has had, their BMI, insulin level, age, and so on.

4.1. Train Scikit-Learn Model

All the codes in this section are in train.ipynb .

Import Packages

import mlflow
from azureml.core import Workspace
import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer
from sklearn.impute import SimpleImputer

Setup workspace

ws = Workspace.from_config()

After running this cell, you might be given an URL to perform web authentication. This is necessary for connecting to Azure Machine Learning workspace. Once that’s done you can return to the IDE and continue with the next step.

Set the tracking URI

An MLFlow tracking URI is an address which we can find the MLFlow tracking server. We set the tracking URI to let MLFlow know where to log the experiments to.

mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())

The tracking URI has the format

azureml://<region>.api.azureml.ms/mlflow/v1.0/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.MachineLearningServices/workspaces/<aml-workspace>?

Set the MLFlow experiment

The code below defines the name of an MLFlow experiment. An MLFlow experiment is a way organizing different runs. An experiment contains multiple runs where each run is an execution of your training code. We can define the parameters, results and artifacts to be stored for each run. If the experiment name does not exist, a new experiment will be created, else it will log the runs into an existing experiment with the same name.

experiment_name = 'diabetes-sklearn'
mlflow.set_experiment(experiment_name)

Load Dataset

input_path = 'pathtodata.csv'

df = pd.read_csv(input_path, sep = ',')

y = df.pop('Outcome')
X = df

X_train, X_test, y_train, y_test = train_test_split(X, y)

Pre-Process

def change_type(x):

    x = x.copy()
    for col in ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'Age']:
        x[col] = x[col].astype('float')

    return x

def replace_zeros(x):

    x = x.copy()
    x[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']] = x[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']].replace(0,np.NaN)

    return x

ft_change_type = FunctionTransformer(change_type)
ft_replace_zeros = FunctionTransformer(replace_zeros)
num_imputer = SimpleImputer()

We create two scikit-learn FunctionTransformer for changing the data types of selected columns to float and replacing zero values with NaN.

Create a scikit-learn pipeline

rf_clf = RandomForestClassifier()
pipe = Pipeline([('change_type', ft_change_type), ('replace_zeros', ft_replace_zeros), ('fillna', num_imputer), ('clf', rf_clf)])

Hyperparameter Tuning

mlflow.sklearn.autolog(max_tuning_runs=None)

param_grid = {'clf__n_estimators': [10,20,30], 'clf__max_depth':[2,7,10]}

clf = GridSearchCV(pipe, param_grid = param_grid, scoring = ['roc_auc', 'precision', 'recall', 'f1', 'accuracy'], refit = 'roc_auc')
clf.fit(X_train, y_train)

The logged results can be found in Azure ML Experiments.

The model artifacts and run_id of the best model can be found in the Outputs + logs tab.

model.pkl is the file that contains the scikit-learn model object. The file path to this model is best_estimator/model.pkl we will need the path for model registration in the next step.
conda.yaml and requirements.txt contains the conda and pip packages required to train the model.
run_id: is the unique identifier to an MLFlow run. We will use it to retrieve the model file in the next step.

4.2. Register Model

The purpose of registering the model to Azure Machine Learning’s model registry is enable users to track changes to the model through model versioning. The following code are written in the register_model.ipynb notebook.

Retrieve the Experiment

We retrieve the experiment from the workspace by defining the workspace and experiment name.

from azureml.core import Experiment, Workspace

experiment_name = 'diabetes-sklearn'

ws = Workspace.from_config()
experiment = Experiment(ws, experiment_name)

Retrieve the Run

Retrieve the run from the experiment using the run_id obtained in the previous section.

run_id = 'e665287a-ce53-41f9-a6c1-d0089a35353a'
run = [r for r in experiment.get_runs() if r.id == run_id][0]

Register the model

model = run.register_model(model_name = 'diabetes_model', model_path = 'best_estimator/model.pkl')

model_name: an arbitrary name given to the registered model
model_path: path to the model.pkl file

We can find the registered model in Azure Machine Learning "Models" tab. Registering a model file to the same model name creates different versions of the model.

We can view the details of the latest version of the model by clicking on the model name. Details include the experiment name and run id which generated the model is useful for maintaining data linage.

4.3. Create Scoring Script

The scoring script typically referred to as score.py is used during inference as the entry point to the model.

score.py consist of two mandatory functions:

init(): loads the model as a global variable
run():receives new data to be scored through the data parameter a. performs pre-processing of the new data (optional) b. performs prediction on the new data c. performs post-processing on the predictions (optional) d. returns the prediction results

# score.py

import json
import os
import joblib
import pandas as pd

def init():
    global model
    model_path = os.path.join(os.getenv('AZUREML_MODEL_DIR'), 'model.pkl')
    model = joblib.load(model_path)

def run(data):
    test_data = pd.DataFrame(json.loads(json.loads(data)['input']))
    proba = model.predict_proba(test_data)[:,-1].tolist()
    return json.dumps({'proba':proba})

4.4. Local Deployment

In this section we debug the webservice locally before deploying to ACI. The codes are written in local_deploy.ipynb.

Define the workspace

from azureml.core import Workspace
ws = Workspace.from_config()

Retrieve the model

Retrieve the registered model by defining the workspace, model name and model version.

from azureml.core.model import Model
model = Model(ws, 'diabetes_model', version=5)

Create custom inference environment

While training the models, we have logged the environment dependencies into MLFlow as a conda.yaml file. We will use this file to create a custom inference environment.

Download the conda.yaml file into your project folder and add azureml-defaults along with any other dependencies that is required during inference under the pip dependencies. Here’s how the conda.yaml looks like now.

# conda.yaml

channels:
- conda-forge
dependencies:
- python=3.7.11
- pip
- pip:
  - mlflow
  - cloudpickle==2.0.0
  - psutil==5.9.0
  - scikit-learn==0.23.2
  **- pandas==1.3.5
  - azureml-defaults**
name: mlflow-env

Next we create an Azure ML Environment named diabetes-env with the dependencies from the conda.yaml file and register it to Azure ML Workspace.

from azureml.core import Environment
env = Environment.from_conda_specification(name='diabetes-env', file_path="./conda.yaml")
env.register(ws)

We can view the registered environment in Azure Machine Learning "Environment" tab under "Custom environments".

Define the inference configuration

Here we define the environment and the scoring script.

from azureml.core.model import InferenceConfig
inference_config = InferenceConfig(
    environment=env,
    source_directory=".",
    entry_script="./score.py",
)

Define deployment configuration

from azureml.core.webservice import LocalWebservice
deployment_config = LocalWebservice.deploy_configuration(port=6789)

Deploy local webservice

Before running the below cell ensure that Docker is running in your local machine.

service = Model.deploy(
    workspace = ws,
    name = 'diabetes-prediction-service',
    models = [model],
    inference_config = inference_config,
    deployment_config = deployment_config,
    overwrite=True)

service.wait_for_deployment(show_output=True)

The model.pkl file will be downloaded from Azure Machine Learning into a temporary local folder and a docker image with the dependencies is created and registered to Azure Container Registry (ACR). The image will be downloaded from ACR to the local machine and a docker container running the webservice is built from the image locally. Below shows the output message of a successful deployment.

We can get the scoring URI using:

print (service.scoring_uri)

>> '<http://localhost:6789/score>'

This is the URI which we will be sending our scoring request to.

4.5. Test Local Webservice

In this section we will test the local webservice. The code are written in inference_test.ipynb.

import requests
import json
import pandas as pd

local_deployment = True
scoring_uri = '<http://localhost:6789/score>'
api_key = None
input_path = 'path/to/data.csv'

# load the data for testing
df = pd.read_csv(input_path, sep = ',')
y = df.pop('Outcome')
X = df
input_data = json.dumps({'input':X.head(1).to_json(orient = 'records')})

if local_deployment:
    headers = {'Content-Type':'application/json'}
else:
    headers = {'Content-Type':'application/json', 'Authorization':('Bearer '+ api_key)}
resp = requests.post(scoring_uri, input_data, headers=headers)

print("prediction:", resp.text)

We sent a post request to the scoring_uri along with the data in JSON format. Here’s how the input_data looks like:

'{"input": "[{\"Pregnancies\":6,\"Glucose\":148,\"BloodPressure\":72,\"SkinThickness\":35,\"Insulin\":0,\"BMI\":33.6,\"DiabetesPedigreeFunction\":0.627,\"Age\":50}]"}'

Here’s a sample of the response for inference on a single record. The return value contains the probability of person being diagnosed with diabetes.

>> "{"proba": [0.6520730332205742]}"

Here’s a sample response for inference on 3 records.

>> "{"proba": [0.5379796003419955, 0.2888339011346382, 0.5526596295928842]}"

The response format can be customized in the run function of the score.py file.

Terminate the local webservice

Terminate the webservice by killing the Docker container using command prompt.

# CLI
docker kill <container id>

4.6. Deploy to Azure Container Instances

After successful testing of the model locally, it is ready to be deployed to ACI. The deployment steps are similar to local deployment. In the aci_deploy.ipynb notebook:

from azureml.core import Workspace
ws = Workspace.from_config()

from azureml.core.model import Model
model = Model(ws, 'diabetes_model', version=5)

from azureml.core import Environment
env = Environment.get(workspace = ws, name = 'diabetes-env', version = 1)

Define the workspace
Retrieve the model from the model registry
Retrieve the environment that we previously registered from the environment registry

Define the inference configuration

from azureml.core.model import InferenceConfig
inference_config = InferenceConfig(
    environment=env,
    source_directory=".",
    entry_script="./score.py")

Define the deployment configuration

from azureml.core.webservice import AciWebservice
deployment_config = AciWebservice.deploy_configuration(cpu_cores=0.1, memory_gb=0.5, auth_enabled=True)

We allocate resources such as cpu_cores and memory_gb to the ACI webservice. When auth_enabled is True the webservice requires an authentication key when the API is called.

Deploy ACI Webservice

service = Model.deploy(
    workspace = ws,
    name = 'diabetes-prediction-service',
    models = [model],
    inference_config = inference_config,
    deployment_config = deployment_config,
    overwrite=True)

service.wait_for_deployment(show_output=True)

The following message will be shown when the deployment is successful:

>> ACI service creation operation finished, operation "Succeeded"

To get the scoring URI:

print (service.scoring_uri)

>> <http://7aa232e8-4b0b-4533-8a84-13f1ad3e350a.eastus.azurecontainer.io/score>

To get the authentication keys:

print (service.get_keys())

>> ('MbrPwtQCkQqGBVcg9SjKCwJjsL3FMFFN', 'bgauLDXRyBMqvL7tBnbLAgTLtLMP7mqe')

Alternatively we can also get the scoring URI and authentication key from Azure Machine Learning "Endpoints" tab.

4.7. Test ACI Webservice

After deploying the model, let’s use inference_test.ipynb again to test the ACI webservice. Change the following parameters and the rest of the code are the same as local testing.

local_deployment = False
scoring_uri = '<http://7aa232e8-4b0b-4533-8a84-13f1ad3e350a.eastus.azurecontainer.io/score>'
api_key = 'MbrPwtQCkQqGBVcg9SjKCwJjsL3FMFFN'

The pricing table for ACI can be found here.