The world’s leading publication for data science, AI, and ML professionals.

Machine Learning Models Deployment

What is it, why does it matter, and its main challenges

Picture from Pixabay
Picture from Pixabay

Introduction

Nowadays, all over the internet, you can find all kinds of resources addressing the science and methodologies to successfully develop a machine learning model. Whether you want to develop a Supervised or Unsupervised Learning model and all its subtypes, thousands of posts will show you how to do it step by step.

However, there is one stage, that is key to bring to life these ML models and really be able to extract value from them: Deploy and productizing them. And this stage is commonly overlooked or not addressed at all.

So, this is what we are going to explore throughout this post: What is model deployment, why does it matter, and what are its main challenges.

Heads up: This article contains affiliate links so that you can comfortably buy this book without any extra charge while contributing to the creation of more posts like this one.

This post is inspired by the great book by Pramod Singh, Deploy Machine Learning Models to Production. If you are interested and want to delve into this topic I strongly encourage you to check it out and buy it.

What is Model Deployment?

I have talked in previous posts about developing a Machine Learning model successfully, and I even posted several practical examples solving problems like regression and classification.

The general process of model development and deployment would look something very similar to this:

Figure by Author, inspired by source
Figure by Author, inspired by source

Building and deploying a model is an iterative process with some cycling parts that would nurture from the information of other steps to be improved. There are some questions that the Data Science team, along with the Business, need to answer:

  • What is the problem that you are trying to solve?
  • How can we use data to solve this problem?
  • What data do we need? And what do we currently have?
  • What are the data sources and how can we access them?
  • Is the data collected representative of the problem?
  • What data preparation is required?
  • How can we visualize data?
  • Does the model developed to answer the initial question or does it need to be adjusted?
  • How can we put the model into practice?
  • How can we obtain constructive feedback into answering the question?

Here, we will focus on the deployment step. Basically, deploying a model is to make it available for other parties to produce value for the business.

We need to deploy at least the stages of:

So, in practice, we do not deploy a Machine Learning model, but a pipeline.

Once the model is ready to be used in a production environment, we need to expose it to unseen data through some APIs. And this is where comes the critical part and the one that presents the challenges that we’ll discuss later.

Steps

The first step is to deploy the ML model in a production environment and test the results.

Then, we should monitor its performance continuously. If the model performs below a certain threshold, it needs to be retrained and this retrained model will substitute the old one.

Through this process, is critical to perform good Model Management of its versions and features so it doesn’t affect previous user requests.

And when this new model is deployed the process would be repeated.

Figure by the Author, inspired by source
Figure by the Author, inspired by source

How ML Applications Differ to Typical Software Applications

Machine Learning Applications are unique and different from other software applications in two main aspects:

  • Underlying model
  • Underlying data

As stated before, deploying the model is basically making it available for other parties to produce value for the business by integrating it into existing business applications.

Simply put, the ML model is exposed as REST API endpoints and serve:

  • Requests within the application platform
  • Requests by direct user

And can be deployed:

  • As a stand-alone predictor using batch data to give outputs
  • As a server of requests in real-time, being a dynamic model

Model deployment is the starting point of the model management phase, which is filled with challenges and potential issues. Therefore, to successfully carry over this phase, there are necessary a lot of inputs and alignment between all the stakeholders involved:

  • Data scientists
  • Data engineers
  • Application developers
  • MLOps and/or DevOps
  • Business team members
  • End users

Why Model Deployment Matters?

We need to understand the role of the ML model in the desired business application. It may have different roles depending on the end goal that this app is serving. It can be a single prediction, a combination of multiple predictions, or some kind of recommendation(s).

A Machine Learning model may be used to predict the clients of a vehicle insurance company that may be potentially interested in obtaining life insurance as well, based on their characteristics (income, age, sex, if they have children or not…).

Besides the possible end goal of the Machine Learning model, its deployment is critical to be able to communicate with the business application and, in the end, to extract value from it. Its predictions would feed the business application and people could benefit from its insights to perform data-driven decision making. Another key point is that predictions can be obtained from real-time data.

Deployment is especially relevant whenever any of the following statements is true: There is a lot of Data, data comes from streaming, there is a big number of users and a fast response is needed.

Main Challenges

The first thing that we should understand is that there are huge differences between developing an ML model and deploying one. These differences translate into huge discrepancies in terms of performance, speed, and resource consumption.

All tasks and priorities should be subordinated to the constant monitorization of the ML model deployed, the application should have the best version to be in production to serve live requests from users.

And, in practice, the role of the ML code in the overall deployment in production is a very small portion of the entire structure needed for it to work correctly. Something in the order shown in the following figure:

Hidden Technical Debt in Machine Learning Systems - Image reproduced by the author
Hidden Technical Debt in Machine Learning Systems – Image reproduced by the author

Stakeholder Coordination

Usually, the most common issue in any Ml Model Deployment is to coordinate the deployment team with other team members who lack machine learning knowledge or understanding.

Model deployment is a group task that needs constant communication and a common understanding of the overall objective. An adequate plan at the early stages of ML model development is key for the MLOps/DevOps team to prepare well for the deployment.

Programming Language Discrepancies

Normally, the ML model is developed in Python or some variation such as TensorFlow or PyTorch, and the application that uses its output is developed in Java, C++, or Ruby. Although nowadays is becoming easier and easier to migrate the ML models to integrate smoothly with the application, programming discrepancy brings some integration challenges to the table.

Model Drift

Model drift is the decrease of the performance of the ML model below the specified benchmark/threshold. This is why is so important to constantly monitor the ML model performance, to be able to detect and update the model to have the best version available online and running.

This performance decrease is often produced by:

  • Changing behavior of the data: due to the dynamic nature of data, it and its inner patterns may change across time. So one ML model performing very well at its deployment will see how its performance decrease across time as the patterns and the adjustment of its inner weights are no longer mirroring current data. In these cases, the model must be retrained with the most actual data and move the new version into production.
  • Changing interpretation of the New Data: This is due to a concept drift and usually, it appears because of a change of the interpretation of a certain label or category. For example, some groups of clients may be characterized to belong to group X and at one point in time, the business decides to change it to class Y. Again, the model will be no longer useful and needs to be retrained and put back into production.

On-Premise vs Cloud-Based Deployment

Both approaches have their own pros and cons and are more suitable depending on the problem trying to solve. So the business, along with the Data Science team, need to analyze and decide which implementation is best.

Clear Ownership

Each member of the team has to clearly know his/her responsibilities and role. Usually, DevOps does not know much about what is going on under the model’s hood and need a deep enough explanation of its behavior to integrate it correctly in the application. On the other hand, Data Scientists tend to assume that their job is done when the model is developed. So it is key the fluent communication across the entire team to ensure that the deployment is done in the required time, cost, and quality and to avoid any potential issues.

Model Performance Monitoring

The monitoring strategy needs to be planned before putting the model into production and normally it will be designed around the following questions:

  • When to retrain the model?
  • How much data will be used to retrain the model?
  • What is the performance threshold for the model?

Version Management

This is a key aspect to consider. It helps to:

  • Track which model is the best
  • The different file dependencies
  • Data resources pointers

Git is the most spread way to track versions and staged deployment.

Privacy-Preserving and Secure Model

This topic is one of the most important ones. It is key to safely secure the model so no adversarial attack could expose its data to third parties.

Conclusion

Throughout this article, we have explored what Model Deployment is, why it matters, and some of the most common challenges it faces. As always, I hope you liked it and learned something new 🙂

If you liked this post, you can look at my other posts on Data Science and Machine Learning here.

If you want to learn more about Machine Learning, Data Science and Artificial Intelligence follow me on Medium, and stay tuned for my next posts!


Related Articles