14 Requirements to Make your Machine Learning Project a Success (Part I)

Set yourself up for success and avoid a disaster

Photo by Andrew Neel at Unsplash

A lot is being written about algorithms and novel solutions, yet not enough is spoken about how to carry out the development of a machine learning project that will add value to your organisation/company. Most of the times a project fails not because of the implementation of the “wrong” algorithm, but due to the lack of organisational support, a clear roadmap and a work methodology that is both straightforward and transparent. Even though this becomes much more critical in start-ups with tight budgets, the complexity of the problem increases in larger organisations where more actors need to be linked.

In this sense, whether you are creating a company from scratch with a product/service based on machine learning, or working in a start-up or a large company with little to no experience in these kind of projects, the objective of this article (divided into two parts) is to offer you a clear picture of how they should be handled to avoid wasting resources, which would in turn also help you to identify poorly organised teams and low quality machine learning external service suppliers that will doom your projects.

Then, what’s the recommended approach for machine learning? For some reason we love lists of top 10 steps/rules but in this case you’ll have to settle with 14. The following list is built based on my personal experience as a Data Scientist and Strategic Consultant, and the valuable stuff I’ve learnt from really insightful leaders. You can see it as a list of requirements that, when not satisfied, are likely to cause your project to fail sooner or later. For the sake of simplicity, the list is divided into 2 sets of requirements, those related to management decisions and those that involve technical reasons that are tightly linked to the development process:

Management requirements

1. Define the problem, the resources needed to solve it and possible limitations
2. Find key actors who have field knowledge of the problem at hand
3. Define the project scope
4. Define success metrics (both technical and business oriented)
5. Set soft/flexible deadlines
6. Give a global picture of how the development will be carried out (a general example that will fit most of the cases is provided further ahead)
7. Find internal champions that will promote your project

Development requirements

1. Understand the data, its sources and generation process
2. Humble yourself up and research new solutions/algorithms
3. Don’t implement models you are not capable of explaining
4. Build benchmark models, fail fast and as many times as possible
5. Discuss your decisions with the whole team as frequently as possible while paying attention to your audience and prioritising transparency
6. Go through the development, testing and production stages
7. Document the whole process (not just the code)

In part I we’ll go over the management requirements.

Management requirements in depth

It does not matter if your role does not involve management, for a project to succeed everyone needs to contribute to its order, even when this is not expected from you. In particular, this becomes much more important when working in technical developments that entail many complex tasks that cannot be carried out masterfully by a single person in a reasonable amount of time. In this regard, the last scenario you want to be in is one where someone without much technical knowledge makes promises that involve accomplishing the impossible, or even worse, reaching the end of the project and realising you made pretty serious mistakes that invalidate the whole development. With this in mind, let’s go over some of the most essential requirements you should try to meet while working on a development that involves machine learning.

1. Define the problem, the resources needed to solve it and possible limitations

Believe it or not, many projects start due to someone saying: Let’s do some machine learning, it sounds cool and will help us in our road of digital transformation. Some may even go even further and say: I don’t care what we do but we need to innovate, AI will make us different from our competitors. Besides these fictional examples, the point here is that machine learning should never be implemented just for the sake of saying you are using it or because it seems innovative. Sounds fair right?

In any case, the first step should always be to start by defining the problem you are trying to solve, the resources needed to do it and the possible limitations (I dare say that you may not even need to use machine learning). Some questions that need to be answered before starting would be:

• What are we trying to solve? Is it relevant? How will this benefit us?
• Has this problem been solved before in other companies/industries?
• How long could a potential development take?
• Do we have enough data to do it? How do we extract it?
• Do we have the infrastructure required to start a development?
• Do we have the team required to go through with this project? If not, how do we build it?
• What is our target/objective?
• Which variables could be used?
• Are there any legal restrictions (for example, GDPR or CCPA)? How would this change our answers to the previous questions?

It is important to note that this requirement should be fulfilled in parallel with requirement number 2, as key actors both with several points of view and a relevant voice in the organisation are needed to answer most of the previous questions.

2. Find key actors who have field knowledge of the problem at hand

Even if you are an expert on the particular field that you are working in, you should always try to identify and involve a set of professionals that will aid you in the understanding of the whole problem, its intricacies and relevant details, from different perspectives. No one knows the problem/business better than the people who live with it. By keeping this in mind you will avoid taking obviously wrong approaches, while building on consensus.

Moreover, remember that even if you’ve solved the problem before or you’ve found a solution developed by a third party, it may not be the same problem once you take into consideration the semantic of the variables that belong to each company/organisation/database. In this sense, two companies can have the exact same database due to the fact that they share the same Finance/HR platform, but the meaning and relevance of each variable can be completely different based on how these platforms are used.

3. Define the project scope

Machine learning projects can take as much time as your creativity lasts. For example, if you ever worked in an ETL process, you should know that you could spend ages trying new imputation methods for missing values (if applicable), working on the detection of outliers/inconsistencies, testing new features/variables, and even looking for new data/external sources.

The question is where do you stop? Surely you can’t keep going on forever as you are expected to show results at some point in time… and the longer you take the longer you’ll have to wait to see some returns on your investment. The answer is to define a project scope that aims to build a minimum viable product (MVP) in a short span of time, and then build on top of it in future iterations of the project. Note: you should be clear and concise about what is included in this MVP so that no doubt can arise by the end of the project.

But what is an acceptable result and how long should you spend trying to achieve it? To answer this we have requirements 4 and 5 (arguably the most challenging ones).

4. Define performance metrics (both technical and business oriented)

Let’s address the two most important aspects of performance metrics: meaning and acceptable values.

Meaning: If your most relevant internal stakeholders cannot explain in simple terms how you measure the performance of your model then you are doing things wrong, as this shows a complete lack of scoping, transparency and consideration for the business needs. In this sense, you should take your time to discuss and explain how you are going to measure the performance. For example, we know that for classification problems, accuracy may be a poor metric, or even misleading when classes are heavily imbalanced. In this example, precision, recall or the f1-score could be some of the better alternatives that can be chosen based on the business objective. Another example would be the time series demand forecasting of products for inventory management, where badly specified models aim to minimise single value loss functions (RMSE, MAE, MAPE, SMAPE, etc.) instead of using a Quantile Loss function that could provide the business with time-varying stock-level policies. If needed, do not be afraid of building custom loss functions.

Acceptable values: you may be tempted to propose acceptable values based on your past experience but, as we know, that is not a wise decision as the quality (consistency and variability) and volume of the available data will tell what is reasonable/achievable. Don’t be that person. First try a simple version of the model you are going to use and check the results, that should be enough to allow you to propose an acceptable value.

5. Set soft/flexible deadlines

Answering how long it will take to finish a project will never be easy (unless the project is really simple). Again, even if you solved the problem before, you might find yourself against several common problems such as:

• bad quality data
• the lack of a data model (or the presence of a messed up one)
• poor variable and process documentation
• slow IT departments that take a lot of time to give you access to the resources you need to work (cloud services for example)
• little to no support from the key actors you’ve identified and so on

To cope with the first three, the best thing you can do is to ask for a couple of days (less than a week) to swiftly go over the data related problems. Once you’ve managed to get a glimpse of the current status of the data you can then propose some soft/flexible deadlines that should vary no more than 2 weeks during the MVP phase. If you are not able to make the first assessment because you are an external consultant/service provider, then you can still provide an answer… Let’s be honest, if you have a capable team of 2/3 data scientists, for most common modelling problems no development of an MVP should take more than 3 months (provided that data engineering tasks are not needed and you are not expected to build a platform). You may be wondering, but what is a common modelling problem? Here are some business oriented examples:

• Demand time series forecasting
• Customer/competitor segmentation
• Recommendation systems
• Text classification
• Sentiment and topic analysis (the simplest and fastest to implement)
• Survival analysis models (customer/employee churn/attrition)
• Price elasticity of demand (done right, not with a simple regression and theoretical distributions)
• Supply chain optimisation (maybe the only one that could take more time to solve but only if it is the first time you are working on it)

Regarding problems that are similar to the last two i.e. delays that involve the proactivity of others, be clear about the expected response times and how its non-compliance will play in the development time.

6. Give a global picture of how the development will be carried out

For the majority of the cases, your projects should follow the same global structure as the one described in the following figures:

Image by Author

You should always aim to first build a prototype by following some standard steps that should be supported by the completion of the previous requirements:

• Defining the problem and scope
• Identifying and extracting the relevant data
• Implementing filters (if applicable), analysing missing data and outliers
• Working on the feature engineering process
• Defining the modelling approach, i.e. under what theoretical framework are we going to be working
• Build a benchmark model and improve it
• Check the performance of the model
• Repeat the whole process until the prototype is good enough according to requirement 4

By showing this workflow you are adding to the transparency of your work and giving a hint about the complexity of the problem to non-data professionals involved in the process. This will in turn help you explain in a clearer manner the possible delays that may occur (specially in the data processing steps).

Once the prototyping stage is cleared, the frequently neglected step that involves moving the final model to production should be addressed. A brief depiction of this process is shown in the next figure.

Image by Author

In short, you need to define the architecture of the solution (nowadays mostly cloud components), organise your code into executable scripts that run on dedicated environments, build a pipeline and orchestrate its execution, build a monitoring process to keep track of the changes in the model performance (beware of data drift), formalise the documentation and, if possible, explore new extensions and identify improvement opportunities.

7. Find internal champions that will promote your project

This last requirement applies only in large organisations. You’ll quickly see why.

The second worst model is the one that is not used. You may have done a terrific job while developing through consensus and using the latest state-of-art algorithms but, if the users don’t see the value of your work you still have one last task to do i.e. make them use it willingly. Here’s where change management strategies come into play, as most of the times the usage of new tools requires some cultural adaptations within the organisation (the larger the worse). The good news is that if you followed the first two requirements i.e. the definition of the problem was discussed and supported by key actors, half of the job is done as there would be additional enforcement coming from the top management. In any case, it is always useful to find internal champions that will promote your projects and enforce their usage within their teams, other teams or even areas of the organisation (maybe you developed a solution that was only intended to be used by a specific area but with some slight changes it could be of help to others).

Concluding remarks

All in all, we’ve gone through some of the most critical requirements that you should consider while approaching a new machine learning project. Many of them may sound obvious to some, but as we know, everything management-related sounds obvious once you read it. At the minimum, this article should either work as: a) a quick reminder of mistakes to avoid for fellow professionals; b) a standard to demand from external/in-house data teams.

As a final advice, always aim/demand for transparency, consensus and quality (if possible don’t rush things). I hope you find value in this short read and stay close for part II.