When it comes to running a Clustering/segmentation project, one of the most challenging tasks is determining how many clusters exist.
The good news is that there are plenty of statistical techniques to try and answer that question, ranging from the elbow method to t-SNE visualisation to the gap statistic.
The bad news is that these techniques are rarely conclusive. The reason machine learning courses use examples such as the Iris flower data set is that the number of clusters is known in advance, and they are quite easy to find.
When you finish studying and start working as a data scientist, you quickly realise that real data is messier, and you have to use a mixture of art and science when the data is not giving you a clear picture.

So when you’re next asked to produce a segmentation, don’t throw away the statistical analysis, just sprinkle in the techniques below to create a clustering that works in practice, not just in theory.
Who’s asking?
Don’t fail before you start; think deeply about who’s asking for the segmentation and what they’re going to do with it.
If it’s the marketing team trying to understand their customers better, then you’re probably going to focus your search between 3 and 10. Why? Because you can’t expect people to carry more than 10 clusters and their profile around in their heads.
If it’s the team that has to come up with new ideas for propositions, you might not need as much concern around how many clusters describe the whole customer base. Instead, you just need to find as many interesting groups as possible which are well-defined enough to spark some new ideas, yet big enough to represent a valuable opportunity.
Or perhaps your segmentation will go directly into a product so customers receive a more personalised experience. In this scenario, you’re less concerned about having a number than humans can remember, you simply want to find as many that prove useful for personalisation, which could be thousands!
Not all preconceptions are bad
It might not be a great strapline for your Data Science team, but you don’t need data for everything.
Stakeholders will have a very good understanding of the business and expectations of the types of clusters that may exist in the customer base.
Whilst it is important to let the data do the talking and not simply produce a clustering that gives stakeholders what they want to hear, it is a useful exercise to understand their expectations in advance for three reasons:
- Validation. If everyone is confident that previous analysis suggests there are customers who only use your service at weekends, it should give you confidence that you’re heading in the right direction if that group starts to appear once you move beyond a certain number of clusters.
- Surprise. It’s unlikely that a single stakeholder will describe all of the clusters you’ll discover in the data, so it becomes a powerful advertisement for data science when you can showcase new insight beyond what they already knew.
- Feature engineering. If you have hypotheses about the clusters that exist, you can revisit your features and make sure you’re feeding rich enough data into your algorithm. How can you expect to find the weekend-only cluster if you have no features capturing the day of the week?
Are your clusters stable?
Let’s say your analysis and use case indicates that an appropriate number of clusters falls between 5 and 8. As with any Machine Learning model, you want a meaningful reflection of your data rather than an anomaly that happened to exist in the data it was trained on.
With this in mind, it’s a useful exercise to see what your clustering would produce based on data taken from different points in time. You may find that the 5 clusters produced are of fairly consistent size and profile (by checking their centroids) over time, whereas 8 is too granular and some of the clusters disappear or take on a different meaning from one month to the next.
In this case, you should favour the clustering that is more consistent and stable.
Are your customers stable?
In the same way that you want your clusters to be stable, you also want the movement between them to make sense.
To make that decision, use the same output from the previous step, but this time check the levels of migration between segments. For example, if 90% of your customers belong to different segments when you run the clustering using January’s data, then February’s, then March’s, alarm bells should be ringing.
You want to find a number of clusters where customers will belong to the same segment if their behaviour is the same, or if they have moved then it is because a change has occurred in their make-up. So double-check how many customers move, but also how those customers have changed to see if you’re generally comfortable with the reason for their re-classification.
Tell me in three words
Finally, get to know your clusters. If you can build an understanding of the customers who belong to each one and why the algorithm grouped them together, you gain confidence that you have an appropriate number.
If your clusters are too broad and hard to define, or there is a group that seems to be a miscellaneous cluster of "everyone else", then you probably don’t have enough of them. Conversely, you may have some clusters to which an extremely small number of customers belong, which could mean you have too many.

A rule of thumb often quoted is the three-adjective rule; if you can describe your clusters in three adjectives then they are well defined and easy to communicate, if you need more, then they may be too granular or hard to interpret. It is only a guideline but proves a useful check if you want the clusters to be known across the business.
To wrap up
Keep in mind the questions below to complement your analysis and gain confidence in your clustering:
- Who is asking for the Segmentation and how will they use it?
- What do stakeholders expect to find in advance?
- Are the clusters the same even if you were to run your algorithm again one month later?
- Do customers generally appear in the same segment over time or move for the right reasons?
- Can you describe each cluster in three adjectives?
Happy clustering!
While you’re here
Feel free to check out my other articles: