A Product Manager’s Guide to Machine Learning: Balanced Scorecard
Research shows that only a fraction of Machine Learning (“ML”) projects will have a business impact¹. In this article, I’ll explore and share my own experiences setting up scorecards to maximize success of ML projects.
While working on the top-line/bottom-line strategy at Amazon, one of my focuses was gaining a working knowledge of designing and building ML models. For example, I’ve learned to build image classification using neural networks. Building ML models and looking at their outcome was absolutely fascinating to me. No matter how fascinating ML is or will become, I currently do not have plans to transition into a Data Scientist or Applied Scientist role. Product Management is my passion but I have always found myself drawn to the math and applications of ML! I was ecstatic when I found a role at Amazon that had a component of leading ML features. I like the sweet spot between Product Management and ML.
“Let’s ML this thing!”
Sharing five lessons that I’ve learned in my role as an ML Product Manager at Amazon:
1. Roles— The first and most obvious lesson is that knowing how to build an ML model is not the same as leading an ML product.
2. ML Projects versus software related projects — Leading a traditional software related project is not the same as leading ML projects. The scientific approach (i.e. observation, hypothesis, test, learn, etc.) still apply to both traditional software and ML projects. However, ML is completely dependent on the quality and quantity of the data that is relevant to the business problem as opposed to traditional software models. Once you overcome the limitations of data, you have loss, optimization, and evaluation intricacies. Did I mention the ambiguity of evaluating the business value sourced and customer value proposition?
3. Level of Complexity — “You just need to launch ML features in this product” is not as simple as it sounds. Understand and respect the years of technical architecture that needs to be readjusted because of ML. The more you dig, the more skeletons you find.
4. Business Impact — Calculating the business impact of an ML model is ridiculously ambiguous. Business needs to “show the money .” As a Product Manager, you have to show the quantitative value. Identifying, understanding, evaluating the right assumptions play a key role. This, of course, needs to be followed by alignment with the broader organization.
5. Customer Impact — Validating the Customer impact with ML could be done without ML. I’ve learned that we can verify the validity and impact of an ML model without ever building one. Work backwards! Build mocks (in various formats) to test the assumptions. Having some validation data boosts the confidence of the organization and helps others rally around the cause.
Tip of the iceberg
So to summarize the lessons I've learned: ML adds a significant amount of uncertainty to business stakeholders. Questions that arise when it comes to launching ML products are: Can it provide a benefit to the business? What evidence exists? What are the assumptions? What are the financial risks? Who is most impacted by it? How complex is the implementation? Is the data used in training the ML model truly representative of production? Do we have sufficient labels? These are just the tip of the iceberg. Many more questions like these will surface throughout the process of launching an ML product.
Building a scorecard
As you see, while leading an ML product, there is a need to simultaneously understand and manage several areas at any given time. During my MBA, I’ve read a case study and an article about the concept of building a balanced scorecard. Almost 30 years ago, Robert S. Kaplan and David P. Norton provided this solution to the problem of managing various dimensions of business². They proposed using a balanced scorecard to look at the business from four important perspectives:
- How do customers see us?
- What must we excel at?
- Can we continue to improve and create value?
- How do we look to shareholders?
“Think of the balanced scorecard as the dials and indicators in an airplane cockpit. For the complex task of navigating and flying an airplane, pilots need detailed information about many aspects of the flight. They need information on fuel, air speed, altitude, bearing, destination, and other indicators that summarize the current and predicted environment.”²
Can we adopt the concept of a balanced scorecard and customize it to manage machine learning products? Here are my scorecards that I’ve developed from my experiences:
1. Data, Data, Data
Yes! Sherlock! I agree that you cannot build bricks without clay. We cannot build ML solutions without the right data. So, let's start with the data, data, data. There is no substitute in understanding and accumulating the right data for your project. Depending on where you are in your project, sit down with your applied scientist to understand the constraints they have with data. Take those concerns seriously and rally your team to address those. Develop a plan to build up the data incrementally. Here is a table that illustrates my data scorecard.
2. Show me the money
ML efforts are tied back to the topline metrics of a business unit. Linking the ML output to the topline is critical to the continued success of the project. When stakeholders express concerns about the potential impact, understand the assumptions that are being questioned and build plans to confirm those assumptions. Collect evidence to show the impact of the model to the stakeholders. Tie the project MoSCoW framework to avoid scope creep even when calculating the impact.
3. Internal Scorecard
Reduce cross-functional ambiguity with clear communication processes. Everybody is invested so over communicate the status of the various milestones in the project.
4. Customer
Customers make a great sounding board to your project, involve them early and work backwards. This should be an obvious step to any product manager, so I won’t bore you with it.
Invitation to you:
Solely focusing on launching the ML solutions without a balanced approach will put you on a circular path. Improve the odds of launching the right solutions with the above mentioned balanced score cards. The list of questions and type of scorecards is ever growing. Please feel free to use these score cards for your projects. I invite you to share other scorecards or relevant questions that were helpful to you. I wish you the best!
Citations:
[1]: Fleming, Reetika, and Phil Ferscht. “How to Avoid Your Looming Machine Learning Crisis.” HFS Research, July 2018. https://1pcll3wzgyqw5kf62erficswwpengine.netdna-ssl.com/wp-content/uploads/2018/07/RS_1807_HfS-POV-MachineLearning-Crisis.pdf.
[2]: Kaplan , & Norton. (1992). The Balanced Scorecard — Measures that Drive Performance . Harvard Business Review , 71–79. Retrieved from https://hbr.org/1992/01/the-balanced-scorecard-measures-that-drive-performance-2
