Notes from Industry
A six-point framework on how to maintain your AI/ML models
As the pandemic has made big changes across our world, we can’t always rely on historical data that we used to train and build our first model versions. We all know — or we should realize by now — that these first versions will break somehow. It is just a matter of time. In our first article from last December, we discussed why you need model monitoring on your AI/ML models. Let’s broaden that discussion, and our vision, by considering a holistic framework to maintain our models. This is critical because models are living, functional tools that support our business decisions, drive revenue, reduce costs, and represent a significant investment by the company. Simply monitoring models is a good start but is not sufficient especially if you have a desire to scale beyond a handful of models in production.
A holistic framework should ensure that your model isn’t biased (we all recall the Amazon hiring model that was only trained using mostly males). It should encompass explainability. It should cover all that we need for full reproducibility when it comes time to retrain. We break it down into six key points as follows.
1. A well-documented purpose is the first step. Our models should align with our business goals and purposes, otherwise they will grow stale and lose potency. This seems obvious, but often is neglected because sometimes modelers get more involved in building them for their research or to satisfy their intellectual curiosity. As my colleague David Bloch said on this blog, “Part of the challenge [is] the difficulty in assessing the value of a good decision.” Sometimes a model can be used to clarify these decisions and more closely map and quantify their value to the business. Having this purpose — and understanding the actual business objectives — moves the model from a data science project and makes it a legitimate part of the business. Part of the purposeful approach is to also think through your goals, KPIs and other metrics to assess ROI and fill out the details about the target end users and delivery mechanisms. Another part is in understanding how a model will be used both before and after it satisfies particular business criteria.
2. Data lineage details. Every model comes with some built-in lineage of its underlying data. The trick is capturing these details and how the data was prepared with sufficient details to ensure that the model can be reproduced and trusted. This is also useful around audit times, so we don’t have to try to track down a model’s data origin story or have to onboard it from scratch.
As I mentioned in my earlier article, even the best models evolve because the underlying data and relationships change over time. Having this data lineage is key to tracking and hopefully preventing concept drift, where the world changes but the model doesn’t reflect these changes. This drift could be caused by changing data distributions, measurements, or the underlying user base that may be ignored by your model. How you document these changes is critical.
3. A full lifecycle tracking system. Like the software development lifecycle, this is a process to link the model runs with specific data versions and is another way to document the various changes made to model elements that were part of the experimental build process. Think what GitHub does for tracking versions of program code or what Docker does to track system definitions and components, or what Kubernetes does for tracking and orchestrating compute versions. As we finish various model runs, we need to be documenting these elements so we can annotate our progress and show how we fixed various problems with our models. The evolution of our models is almost more important than the actual models themselves because we can better understand what we are modelling and why we chose not only to build them in the first place but adjust their data inputs and assumptions.
4. A model registry that links to the lifecycle tracking system mentioned in the point above. The registry can also be used to track the model version history where each version is fully reproducible with the same elements as our experiments in changing data, code, software, and hardware platforms. The ideal situation is to have a central registry with a summary dashboard where you can browse the model versions and drill down into each one’s history.
5. Validation routines that document code reviews, reports on the various explanations about its ethical and bias checks and obtains the stamp of approval from its users. This would also be a good place to report on its service level agreements and other functional tests we’ve done and comments on its general production readiness. I have seen many modelers who skip this step. Validation is key to making sure that the model actually does what you intended it to do. It is also the key to making a decision when the useful life of a model is nearing its end and needs to be retired or rebuilt.
6. The last point is having an open model monitoring system. This is what I discussed in my December post, and should be used to capture items such as data drift, a single ground truth, measurement accuracy and provide drill-down capabilities to troubleshoot signals. The monitoring system should also be able to detect anomalies and automatically alert stakeholders when certain thresholds are exceeded.
As you expand your investment in data science and modeling, you will need to manage and maintain an ever increasing collection of models that your business depends on daily. Here are two ways you can get started. First, review your current model maintenance plan against each of the six objectives noted here. Second, create a task force for the effort, or consider getting external help. It will take an up-front investment of time and resources, but you’ll end up with better models that live longer, are safer, and play a larger role in guiding your business decisions. The businesses that get ahead of the curve with model care will be strongly positioned for competitive advantage for years to come. Think of this resource as a way to envision and guide a solid future for your business.
