The world’s leading publication for data science, AI, and ML professionals.

Don’t Let a Machine Learning Engineer Work Alone

Partner with these two key roles for success

Image under license to Mary Alfheim from Shutterstock
Image under license to Mary Alfheim from Shutterstock

Machine learning engineers are in enormous demand. You already are painfully aware of this if you’ve tried to hire anyone with a title even close to this in the past few years. These engineers are incredibly skilled at taking a Data Science output, like a model, and actually putting it into production to ship new applications. Hiring MLEs may represent an enormous investment for your organization, and you may be tempted to assume you don’t need to spend anywhere else to achieve success. Unfortunately, you’d be wrong.

Let’s say you are responsible for a new feature that shows recommendations for a retail product, song, video, or even a job listing to users of your digital product. Perhaps you are a data science or AI/ML lead, or a product manager for a data driven feature. You have carefully worked with customers or other product managers to define what you want to build, and scaled your data science team to deliver an initial recommendation model. Making this model scale to real time delivery, and exposing new recommendations to a user in your product looks like the next big hurdle. Lucky for you, the machine learning engineers on your team are able to scale this solution, and build the pipeline to show the right recommendations to the right user. Done, right?

Nope.

While a great data science and machine learning team will have already trained and tested the model, and evaluated its precision using something like a confusion matrix, you still don’t know how this recommendation will work in the wilds of your actual product. Will users react to this as expected? Are there errors we didn’t anticipate? Is this actually driving the outcomes we hypothesized it would? How can we make this better?

Meet the visualization engineer

You need to make sure that your team is also equipped with a visualization engineer. The person in this role will provide actual tracking or reporting against observed user behaviors related to your recommendation, or whatever your user facing feature is, once it is in a customer facing UI.

This engineer will work closely with your MLEs to understand the feature and model intention, its deployment dates, and its expected outcomes. They will then use your event data to build dashboards that can visualize actual user behaviors. Use this for actual analysis – do you see any unexpected behaviors, what does adoption look like, and what kinds of actions are users taking?

Don’t forget to experiment

You also need to put those observed, real life interactions with your machine learning feature in context. Ideally, you should expose your new feature to a randomized set of users, and suppress other groups from seeing it at all. You can then measure outcomes at the group level – revenue, retention, sales, customer lifetime value, or other metrics that are relevant to your business – by who could use or has seen the new feature, and who has not. A visualization engineer can also put this kind of business intelligence into a dashboard, and clearly indicate patterns of behavior between the groups. Use experimentation, and its results, to put some measure of actual value around the work and the experience that you have delivered. Use this to help inform new investments, and make sure you are driving value for the customer.

You will also want to know whether the algorithm itself powering your feature is the best it can be. In addition to just testing feature/no feature groups, train newer versions of the model, or try new methodologies all together. Deploy both (or more) versions at once in an A/B test, and work with your visualization engineer again to measure and describe the differences in behavior and in outcomes between groups. Identify a new champion model and rely on the MLE to deploy to production for all users. You can repeat this cycle as often as you can create new challenger models to test.

Putting the team together

A Machine Learning engineer, visualization engineer, and experimentation lead can all work together to complete a lifecycle for a machine learning product. Without observing your feature in production, and experimenting around its methodology, you won’t be able to improve your current work or make sure you are delivering value for your users.

It can be very difficult to recruit and retain machine learning engineers. If you are able to do this, be sure to power the team up with visualization and experimentation capabilities to see the AI/ML project all the way through.


Related Articles