The world’s leading publication for data science, AI, and ML professionals.

Lean Machine Learning: Running your Proof of Concept the Lean Startup Way (part 1)

How to transform the way you build machine learning products

Figure 1: Build Measure Learn cycle in The Lean Startup by Eric Reis. Image by author.
Figure 1: Build Measure Learn cycle in The Lean Startup by Eric Reis. Image by author.

In his book The Lean Startup, Eric Ries describes his approach to building a successful startup using the Build Measure Learn cycle depicted in Figure 1. While I don’t own a startup, I’ve been able to apply many of Ries’ ideas in my life as a Data Scientist.

Throughout this series, I’ll describe how to make better machine learning products by applying the Lean Startup approach. In part one, I’ll introduce the Lean Startup Way, the four key areas of the innovation sweet spot, and how to determine if your machine learning product is desirable.

Introduction to the Lean Startup Way

Too often, I’ve seen Data Scientists spend months in a silo building Machine Learning models with incredibly high accuracy. They’re extremely proud of their work, but while presenting their results, they often find that the business customers don’t really understand or care about the models they’ve spent months on. Before diving into building a machine learning model, I recommend Data Scientists follow the Build Measure Learn cycle and first determine what they want to learn from their model.

One of the most critical points Ries describes in the book is the difference between execution and planning. When creating a product, it can be tempting to first build the product, then measure its success, and finally analyze what you learned once the product hits the market. However, Ries argues that the steps for creating a successful product should be reversed.

Figure 2: How to plan your product the Lean Startup Way. Image by author.
Figure 2: How to plan your product the Lean Startup Way. Image by author.

You should first determine what you want to learn. You’re ultimately trying to learn what is the ideal product for the customer and the problem you’re trying to solve. Once you know what you’re trying to learn, your next step is to figure out how to measure the right things so that you can capture what you wanted to learn. It’s only after you know what to measure that you can plan how you will build it. Throughout my career, I’ve seen teams immediately start building but wait to capture metrics until the product was completely built and released. What they found is that the way they built the product didn’t allow them to collect the data we would need to capture what we want to learn.

As such, it is important to ensure you build your product in a way that can actually capture the data you need to build your metrics and not some last-minute, thrown-together "vanity" metric.

The Innovation Sweet Spot

In order to apply the Lean Startup to machine learning, you should start by planning your proof of concept (POC). You already know that you’re trying to learn, but what are you trying to learn? To answer this, you can reference the innovation sweet spot (ISS), a popular concept in Product Management illustrated in Figure 3.

Figure 3: The Innovation Sweet Spot. Image by author.
Figure 3: The Innovation Sweet Spot. Image by author.

Traditionally, the ISS is composed of three key areas: Desirable, Feasible, and Viable. A great product idea is one that is desirable, feasible, and viable. I’ve found one key area to be missing from the ISS – Ethics. A great product, especially one involving machine learning, should also be Ethical. Too often, ethics are overlooked in the machine learning community, so I’ve added Ethical to the ISS depicted in Figure 3.

But how can you apply the ISS to a machine learning product? Let’s take a look at each area of the ISS.

Innovation Sweet Spot – Desirable

Beginning with desirable, you need to ensure that your users want your product. For example, does a Netflix user want the platform to recommend television shows and movies to them, or do they actually enjoy searching for content on their own? It’s probably not a good use of your time to build something that your customer doesn’t want. You shouldn’t write a single line of code until you establish some sense of desirability.

Innovation Sweet Spot – Feasible

Technical feasibility is oftentimes what people focus on the most during their POC. Can my machine learning model achieve 90% accuracy? Is the data quality high enough to train my model? In this area, we want to determine if it’s technically feasible to solve our problem with machine learning. While many Data Scientists assess technical feasibility with metrics like accuracy, precision, and recall, I’ll cover the importance of assessing other metrics like the risk of an incorrect prediction in part 2. If you choose the wrong metrics for determining Feasibility, you will be walking yourself into a painful situation.

Innovation Sweet Spot – Viable

For many Data Scientists, determining viability is something that is often left to the product manager, who would say that a viable product is one that produces more value than it costs to build and maintain. However, Data Scientists play a large role in how expensive a solution is to maintain due to the decisions they make. For example, a deep learning model trained via GPUs will likely be more expensive to maintain than a simple logistic regression model. If you’re not careful, you could easily end up with a non-viable product.

Photo by Giorgio Trovato on Unsplash
Photo by Giorgio Trovato on Unsplash

Innovation Sweet Spot – Ethical

Finally, we hit the most important of them all – data ethics is a serious issue. Thankfully, many people have been shining a light on this area in recent years. Additionally, many institutions now have courses on data ethics in machine learning. While I’ll try to do this topic decent justice in part 3 (coming soon!), I really can’t stress how important it is to establish a data ethics policy and create an advisory board for machine learning in your organization.

Many people think that a POC has failed if the product isn’t implemented. However, a POC is an experiment, and the goal of an experiment is to learn. When you begin, you don’t know if your product will work. If you find out that it doesn’t work, you just learned a product idea that is not worth spending further resources to implement. Congrats, you were successful. Seriously! Just make sure you set yourself up to learn the right things. Ultimately, if we haven’t captured some learnings around each area of the ISS – Desirable, Feasible, Viable, and Ethical – our POC has failed.

Deep Dive on Determining Desirability

In conversations with fellow Data Scientists, I’ve often been asked, "When do you start learning if your customer wants your machine learning product?". I always recommend that this is actually the first thing you do when interfacing with your customers.

Determining if a machine learning product is desirable wasn’t something I was taught in school (check out this article for more on that), but over the years I’ve picked up a few techniques from my colleagues in other departments. If you have strategies to gauge desirability, please drop them in the comments for all of us!

Figure 4: Vitamins vs Painkillers. Image by author.
Figure 4: Vitamins vs Painkillers. Image by author.

A product manager once told me that when you’re assessing potential products to build, you should determine if the product is a vitamin or a painkiller. Vitamins are great to help our bodies function properly, but they oftentimes are not essential with proper diet and exercise. In contrast, if you’ve ever experienced a critical injury or surgery, you’ve likely found painkillers to be essential during your recovery.

I’ve seen a lot of customers in pain whether they realized it or not. Many people become numb to processes that could be improved because "that’s just the way we do things". If you present a vision of "how life could be", you’ll see right away if you’ve found a Painkiller.

So what does presenting a vision of "how life could be" look like? Well, in my experience, it can take many different forms. The simplest form is a presentation to your customers. Upon giving the presentation, I will provide a survey where I can collect metrics. And yes! You just experienced a Lean Startup. I decide what I’m trying to learn – is this desirable? I then decide what metrics I can build to learn this. And finally, I build a presentation (the product) with a survey built into it, so I can capture the metrics.

Figure 5: An example of capturing desirability with a presentation. Image by author.
Figure 5: An example of capturing desirability with a presentation. Image by author.

But is a presentation the only way to determine desirability? Nope – we can pull from other fields like User Experience (UX) by creating a mockup or wireframe. This mockup is a visual, and ideally interactive, representation of what the product could be. UX designers usually use tools like Balsamiq and Axure for this, but I’ve seen Software Developers use tools like Microsoft Visio, Excel, PowerPoint, and Keynote. In fact, with the right packages installed, Keynote allows you to simulate a real website or mobile device.

These tools all come with various levels of sophistication, but the point is you can put together a decent mockup in an afternoon to figure out how your customer feels about your product idea.

You might be wondering, "How the heck do you create a mockup for machine learning?". The funny thing is that your mockup usually doesn’t even require any machine learning! In your mockup, you can begin by building a very naive and fast rules-based model. Your customer likely doesn’t expect you to completely solve the problem by your initial meeting, and building a fast, rules-based model will give them a feeling of "how life could be".

I’ve even produced machine learning mockups to classify documents using a dictionary of a few keywords – no machine learning required! When the customer was able to drag their files into one folder, and the documents were magically moved into the appropriate folder, they were immediately sold on funding the product. Never mind that if they tried to cover every edge case it would have failed spectacularly! The point is they were easily able to get the vision, and I was able to easily tell if it was a painkiller or a vitamin.

Figure 6: A simple mockup for measuring the desirability of a machine learning classifier that automatically classifies documents by moving them into folders. Image by author.
Figure 6: A simple mockup for measuring the desirability of a machine learning classifier that automatically classifies documents by moving them into folders. Image by author.

A more sophisticated version of a mockup might involve building a website for your product and advertising online via Google, Facebook, or some other high-traffic site. After creating your website and ads, you can take one of two approaches to determine desirability.

One approach is to capture individuals’ contact information, usually in the form of an email address. This may capture if an individual has a general interest in your product, but it doesn’t tell you if they desire the product enough to pay for it. It’s critical to determine if consumers will spend their hard-earned dollars on your product.

The second approach to determining desirability is to allow individuals to pre-order your product. When a customer lands on your website, you can allow them to pre-order and fully pay for the product in advance, or allow them to place a small deposit with their pre-order. However, if the deposit is too small, some individuals may cancel before the product is complete. Ultimately, having a large pre-order list can stoke interest from investors.

Conclusion

When creating a machine learning product, it’s incredibly important to first determine if your product is desirable. You should try to measure desirability in a way that will produce a clear and strong signal, but your mileage will vary depending on your customer and where they reside. For example, if you’re building a machine learning product that is internal to your company, you probably won’t be able to measure desirability by building a website that asks for employees to place a deposit.

And don’t forget – you’re beginning with an experiment. You’re not looking for a high-fidelity product right away. Try to capture your learnings as quickly as possible. Just remember to first determine what you want to learn, then decide how to measure it, and finally, decide what you need to build in order to capture those metrics.

References

  1. Eric Ries, The Lean Startup (2011), http://theleanstartup.com/book

Related Articles