Seven Questions to Ask before Introducing AI into Your Project
AKA First rule of machine learning? Don’t use machine learning.
With all the buzz around the success of machine learning in finding new drugs and empowering self-driving cars the expectation is that it is available and very affordable. If it can solve those big problems it must be easy to apply to something smaller.
I remember the excitement during my undergrad when I first learned about CNNs. In young boisterous minds all problems had a solution and this solution was deep neural networks. Experience taught me otherwise — if not used properly, machine learning algorithms turn into a problem themselves.
While writing the article I stumbled upon this LinkedIn post drawing the difference between two machine learning teams approaching the same problem — one rushing to bring a machine learning solution at all costs and another putting time into understanding the problem. I am a much bigger fan of the second approach.
This article is a summary of areas that I pay attention to while deciding whether to go ahead with a machine learning solution. With all that said, let’s dive in!
Problem
Is the problem suitable for machine learning?
Machine learning can be viewed from multiple angles, and in a simplified world it is a tool that learns patterns from the training data useful for making predictions on unseen test examples.
Naturally, patterns in the data can be simple enough to be cracked with a set of rules or an analytic solution. For example, forecasting the next sunrise time could easily be done with an analytic solution guided by a set of physical rules. In these cases a machine learning solution would be an overkill.
At the same time, patterns might be too complex for the current models. Such is a machine-translation problem, where to go beyond simple dictionary mapping and achieve human-level performance a huge corpus of wide information about the world needs to be incorporated into the model.
A sweet-spot problem would be a very specific task with a fixed set of output options and a clear objective, where there is a clear pattern that can be learned — image or text classification, image enhancement, speech to text mapping.
Data
Is there enough data? Is the data of reasonable quality?
Perhaps the major obstacle to developing high-quality solutions is the lack of a sufficient amount of high-quality data. For many problems the data needs to be mined — creating a market for data mining tools, such as Amazon Mechanical Turk. Ideally, the data needs to be balanced, versatile, heterogeneous and well labeled.
If the dataset consists of 1000 images of cats and ten images of dogs, it is very likely that the plain model wouldn’t be able to capture the distribution of dogs. In this case models that can be trained on unbalanced data are usually used. The data needs to have a good selection of cat and dog breeds for the model to learn common features. Dogs mislabeled as cats would very likely confuse the model.
Cost
Can both development and maintenance costs be sustained?
Development cost
There are several aspects that are very costly in developing machine learning solution — team, hardware, and data.
As in many problems, the people are crucial to success. Top machine learning engineers have high level degrees with their skills in high demand, meaning that their cost will be exceptionally high. An average UK salary for a ML Engineer is 60k£.
People are not the only expense — machine learning models need to be trained on specialized hardware. Startups often get hundreds of thousand of credits from tech giants to use on hardware. However, it is rarely the case for established businesses, meaning that they would need to pay for it themselves.
Finally, data… If we don’t have access to free data, it would need to be mined. A single response on Amazon Mechanical Turk, for a very simple task of a single click labelling, would cost around 0.05$. To get a well labelled dataset with 1,000,000 images we would have to pay $50k. And this is with a very simple, one-click task. For more complex problems the cost could easily go beyond $100k.
Maintenance cost
Machine learning models need maintenance to ensure that the model that drives your decisions is up to date with the current data distribution. That means that you would ideally need an engineer to maintain it. Top products usually stay in the iterative feedback cycle.
The hardware on which the model is to be deployed (e.g. cloud) would also need to be paid for. And the cost would sky-rocket with the business scaling up. Workload distribution could be challenging — not enough machines would mean delays in processing the request. Too many machines and the cost would go up.
Development time
Can you afford at least half a year for solving a problem?
If the solution is not available out of the box, such as services provided by Google/Amazon/Microsoft, the development and deployment time can be quite long. For long term solutions you need to have data, training and testing pipelines established.
Development is an iterative process. From my personal experience a month is required to set up a very simple model. Data collection and cleaning performed in parallel would take roughly the same amount of time. Simple training and deployment pipeline would also take around a month.
Complexity is usually added on top iteratively — with every step the model the data pipeline are becoming more sophisticated and thus would need to be actively maintained. Depending on the application there could be dozens of iterations before the solution is actually reached.
Explainability
Is knowing how the algorithm arrived at the solution essential for the application?
Although not that important for many non-life-critical applications, understanding how ML algorithms arrive at conclusions could be useful. For example, drug discovery would greatly benefit from explainability as we would get hints on the patterns guiding successful discoveries.
In life-critical situations explainability is essential —in diagnosing diseases with machine learning models it is essential to know how and why the model arrived at the conclusion to make sure that the patient is not mistreated. Understanding how self-driving cars make decisions is also important, to ensure that no spurious correlations are learned that could result in accidents on the roads.
Ethics
Is your model making decision that could discriminate the end users?
The infamous twitter bot that within a glimpse turned into a racist prick is a good, but probably rarely explicit example of how a model can make a wrong turn. Many of the models employed in practice have a latent bias which is often very hard to identify.
In some cases, this bias might not be such a big deal — for example a chat-bot that learned to be nicer during the afternoon (perhaps trained on a dataset with time-stamps — people do tend to be slightly nicer after they have lunch). Whereas a chat-bot that is friendlier with women than with men would be quite worrying but hard to diagnose problem. Typically biases point to serious problems in the training data.
Processing time
Does the application need to run in real time?
Many machine learning algorithms, due to their complexity, would struggle with achieving time performances required by some applications. Heavy models that do perform well in terms of accuracy would require a substantial amount of computation and memory.
Two examples where machine learning would be challenging to implement are high-frequency trading and computer graphics. In these two cases speed is so important, that simple heuristics are usually employed.
Mobile phones is another example where implementing a machine learning solution could be challenging due to rather scarce resources. That is the reason why many mobile phones are now introducing Neural Processing Units (NPUs) — enabling models to be put on mobile devices.
Conclusion
There is a reason why undergraduate students in their final year are discouraged to do machine learning projects — it is simply too risky. Universities have gone through all the pitfalls before learning that it makes the project exposed to obvious risks.
Hearing that from a Machine Learning Engineer might sound like I am an insurance agent that persuades you not to buy into his services. But in this case honesty is the best policy. After all it is very easy to burn tons of money while following the hype that is actually not worse it.
For those brave enough to start on an ML project, check out my recent article on the tools I found particularly useful for managing AI projects.
If you liked this article share it with a friend! To read more on machine learning and image processing topics press subscribe!
Further reading
I am a great fan of Cassie Kozyrkov’s set of articles on ML use-cases. Those are my favourite:
Do you actually need machine learning in your project? Advice for finding AI use cases
A more detailed checklist of questions to consider before starting on an AI project: Is your AI project a nonstarter?
A more detailed guide on how on the sanity checks before starting an AI projects and wide brushes on what to look for in going ahead with one: Getting started with AI? Start here!
You might as well want check out googles guide for ML Engineers.
Liked the author? Stay connected!
Have I missed anything? Do not hesitate to leave a note, comment or message me directly on LinkedIn or Twitter!
