The Machine Learning Lifecycle in 2021
How do you actually complete a machine learning project and what are some tools that can help each step of the way?
Everyone and their mother is getting into machine learning (ML) in this day and age. It seems that every company that is collecting data is trying to figure out some way to use AI and ML to analyze their business and provide automated solutions.
The machine learning market cap is expected to reach $117 billion by 2027 — Fortune Business Insights
This influx of popularity in ML is leading to a lot of newcomers without a formal background getting into the space. While it’s great that more people are getting excited and learning about this field, it needs to be clear that incorporating an ML project in a production setting is not an easy task.
55% of businesses working on ML models have yet to get them into production — Algorithmia
Many people seem to be under the assumption that an ML project is fairly straightforward if you have the data and computing resources necessary to train a model. They could not be more wrong. This assumption seems to lead to significant time and monetary costs without ever deploying a model.
In this article, we’ll discuss what the lifecycle of an ML project actually looks like and some tools to help tackle it.
The Machine Learning Lifecycle
In reality, machine learning projects are not straightforward, they are a cycle iterating between improving the data, model, and evaluation that is never really finished. This cycle is crucial in developing an ML model because it focuses on using model results and evaluation to refine your dataset. A high-quality dataset is the most surefire way to train a high-quality model. The speed that this cycle is iterated through is what determines your costs, luckily there are tools that can help speed up the cycle without sacrificing quality.
Much like any system, even a deployed ML model requires monitoring, maintenance, and updates. You can’t just deploy an ML model and forget about it, expecting it to work as well as it did on your test set in the real world for the rest of time. ML models deployed in production environments are going to need updates as you find biases in the model, add new sources of data, require additional functionality, etc. This brings you right back into the data, model, and evaluation cycle.
As of 2021, deep learning has been prominent for over a decade now and helped bring ML front and center in the market. The ML industry has undergone a boom with countless products being developed to aid in the creation of ML models. Every step of the ML lifecycle has some tool that you can use to expedite the process and not end up as one of the companies with an ML project that never sees the light of day.
The next sections will deep dive into each phase of the ML lifecycle and highlight popular tools.
Phase 1: Data
While the end goal is a high-quality model, the lifeblood of training a good model is in the amount and more importantly the quality of the data being passed into it.
The primary data related steps in the ML lifecycle are:
Data Collection — Collect as much raw data as possible regardless of quality In the end, only a small subset of it will be annotated anyway which is where most of the cost comes from. It is useful to have a lot of data available to add as needed when problems arise with model performance.
Define your annotation schema — This is one of the most important parts of the data phase of the lifecycle, and it often gets overlooked. A poorly constructed annotation schema will result in ambiguous classes and edge cases that make it much more difficult to train a model.
For example, the performance of object detection models depends heavily on attributes like size, localization, orientation, and truncation. So including attributes like object size, density, and occlusion during annotation can provide critical metadata needed to create high-quality training datasets that models can learn from.
- Matplotlib, Plotly — Plot properties of your data
- Tableu — Analytics platform to better understand your data
Data Annotation—Annotation is a tedious process of performing the same task on and on for hours at a time, which is why annotation services are a booming business. The result is that annotators will likely make numerous mistakes. While most annotation firms guarantee a maximum error percentage (ex. 2% max error), a larger problem is a poorly defined annotation schema resulting in annotators deciding to label samples differently. This is harder to spot by the QA team of an annotation firm and is something that you need to check yourself.
- Scale, Labelbox, Prodigy — Popular annotation services
- Mechanical Turk — Crowdsourced annotation
- CVAT — DIY computer vision annotation
- Doccano — NLP specific annotation tool
- Centaur Labs — Medical data labeling service
Improve dataset and annotations — You will likely spend the majority of your time here when trying to improve model performance. If your model is learning but not performing well, the culprit is almost always a training dataset containing biases and mistakes that are creating a performance ceiling for your model. Improving your model generally involves things like hard sample mining (adding new training data similar to other samples the model failed on), rebalancing your dataset based on biases your model has learned, and updating your annotations and schema to add new labels and refine existing ones.
Phase 2: Model
Even though the output of this process is a model, you will ideally spend the least amount of time in this loop.
Explore existing pretrained models — The goal here is to reuse as many available resources as possible to give yourself the best head start to model production. Transfer learning is a core tenant of deep learning in this day and age. You will likely not be creating a model from scratch, but instead fine-tuning an existing model that was pretrained on a related task. For example, if you want to create a mask detection model, you will likely download a pretrained face detection model from GitHub since that is a more popular topic with more prior work.
- FiftyOne model zoo — Download and run models in one line of code
- TensorFlow Hub — Repository of trained ML models
- modelzoo.co — Pretrained deep learning models for various tasks and libraries
Construct training loop — Your data will likely differ in some way from what was used to pretrain the model. For image datasets, things like input resolution and object sizes need to be taken into account when setting up the training pipeline for your model. You will also need to modify the output structure of the model to match the classes and structure of your labels. PyTorch lightning provides an easy way to scale up model training with limited code.
- Scikit Learn — Build and visualize classic ML systems
- PyTorch, PyTorch Lightning, TensorFlow, TRAX — Popular deep learning Python libraries
- Sagemaker — Build and train ML systems in the Sagemaker IDE
Experiment Tracking — This entire cycle will likely require multiple iterations. You will end up training a lot of different models so being meticulous in your tracking of different versions of a model and the hyperparameters and data it was trained on will help a great deal to keep things organized.
- Tensorboard, Weights & Biases, MLFlow — Visualize and track model hyperparameters
Side Note: Even if you think your task is completely unique, here are some pre-training techniques to consider. I would recommend looking into ways to pretrain your model in unsupervised or semi-supervised ways, still only using a small subset of your total raw data for finetuning. Depending on your task, you could also look into synthetic data to pretrain your model. The goal is just to get a model that has learned a good representation of your data so that your fine-tuning dataset only needs to be used to train a few layers worth of model parameters.
Phase 3: Evaluation
Once you managed to get a model that has learned your training data, it’s time to dig in and see how well it can perform on new data.
The key steps for evaluating an ML model:
Visualize model outputs — As soon as you have a trained model, you need to immediately run it on a few samples and look at the output. This is the best way to find if there are any bugs in your training/evaluation pipeline before running evaluation on your entire test set. It will also show if there are any glaring errors, like if two of your classes have been mislabeled.
- OpenCV, Numpy, Matplotlib — Write custom visualization scripts
- FiftyOne — Visualize outputs of computer vision tasks on images and videos
Choose the right metric — Coming up with one or a few metrics can help in comparing the overall performance of models. In order to make sure you pick the best models for your task, you should develop metrics in line with your end goal. You should also update metrics as you find other important qualities you want to track. For example, if you want to start tracking how well your object detection model performs on small objects, use mAP on objects with a bounding box < 0.05 as one of your metrics.
While these gross dataset metrics can be useful in comparing the performance of multiple models, they rarely help in understanding how to improve the performance of a model.
- Scikit Learn — Provides common metrics
- Python, Numpy — Develop custom metrics
Look at failure cases —Everything your model does is based on the data that it was trained on. So assuming that it is able to learn something, if it is performing more poorly than you would expect, you need to take a look at the data. It can be useful to look at cases where your model is doing well, but it is vital to look at false positives and false negatives, where your model predicted something incorrectly. After looking through enough of these samples, you will start to see patterns of failure in your model.
For example, the image below shows a sample from the Open Images dataset, one false positive is shown as the back wheel. This false positive turns out to have been a missing annotation. Verifying all wheel annotations in the dataset and fixing other similar mistakes can help improve your model’s performance on wheels.
- FiftyOne, Aquarium, Scale Nucleus — Dataset debugging to find mistakes
Formulate solutions — Identifying failure cases is the first step in figuring out ways to fix improve your model performance. In most cases, it goes back to adding training data similar to where your model failed but it can also include things like changing pre- or post-processing steps in your pipeline or fixing annotations. No matter what the solution is, you can only fix the problems with your model by understanding where it fails.
Phase 4: Production
Finally! You’ve got a model that performs well on your evaluation metrics with no major errors on various edge cases.
Now you’ll need to:
Monitor model — Test your deployment to ensure that your model is still performing as expected on test data with respect to your evaluation metrics and things like inference speed.
- Pachyderm, Algorithmia, Datarobot, Kubeflow, MLFlow — Deploy and monitor models and pipelines
- Amazon Web Services, Google AutoML, Microsoft Azure — Cloud-based solutions for ML models
Evaluate new data — Using a model in production means you will frequently pass brand new data through the model that it has never been tested on. It’s important to perform evaluation and dig into specific samples to see how your model performs on any new data it encounters.
Continue understanding model — Some errors and biases in your model can be deep-seated and take a long time to uncover. You need to continuously test and probe your model for various edge cases and trends that could cause problems if they were to be discovered by clients instead.
Expand capabilities — Even if everything is working perfectly, it’s possible that the model isn’t increasing profits as much as you hoped. From adding new classes, developing new data streams, and making the model more efficient there are countless ways to expand the capabilities of your current model to make it even better. Any time you want to improve your system, you will need to restart the ML lifecycle to update your data, model, and evaluate it all to make sure your new features work as expected.
FiftyOne
The above is pretty general and unbiased, but I want to tell you a little bit more about the tool I’ve been working on.
Lots of tools exist for various portions of the ML lifecycle. However, there is a pretty glaring lack of tools that help some of the key points I’ve stressed in this post. Things like visualizing complex data (like image or video) and labels or writing queries to find specific cases where your model performs poorly are generally done through manual scripting.
I have been working at Voxel51 developing FiftyOne, an open-source data visualization tool designed to help debug datasets and models and fill this void. FiftyOne lets you visualize your image and video datasets and model predictions in a GUI either locally or remotely. It also provides powerful capabilities to evaluate models and write advanced queries for any aspects of your dataset or model output.
FiftyOne can run in notebooks so try it out in your browser with this Colab Notebook. Alternatively, you can easily install it with pip.
pip install fiftyone
Summary
Only a fraction of all companies that try to incorporate machine learning (ML) into their business manage to actually deploy a model to production. The lifecycle of an ML model is not straight forward but requires continuous iterations between data and annotation improvements, model and training pipeline construction, and sample-level evaluation. If you know what you’re in for, this cycle can eventually lead to a production-ready model, but it will also need to be maintained and updated over time. Luckily, there are countless tools developed to aid in every step of this process.
