Precision & Recall: Explained by Men In Black
An explanation you won’t forget (even if you’re neuralyzed!)
Disclaimer: All opinions expressed are my own.
I don’t know about you but every time I come across the concepts of precision and recall, I understand them completely….and then, all of a sudden the next day I have trouble explaining them. It is erased from my memory as though I had been subjected to a neuralyzer!
So, inspired by the scene where Will Smith’s character passes the MIB exam by shooting a little girl, I came up with an easy-to-understand example that can help me comprehend and remember the concepts of precision and recall. Read on!
Contents
- Recall
1.1 🥱Boring definition
1.2 👽Interesting (contextual) definition
1.3 📈What does high recall mean?
1.4 📉What does low recall mean?
1.5 💵A real world example that makes c̶e̶n̶t̶s̶ sense - Precision
2.1 🥱Boring definition
2.2 👽Interesting (contextual) definition
2.3 📈What does high precision mean?
2.4 📉What does low precision mean?
2.5 💵A real world example that makes c̶e̶n̶t̶s̶ sense - F1 Score
- Why not keep things simple and just use accuracy?
- Final Takeaways
0. Running Example
You are an agent of the Men In Black (MIB), a secret agency responsible for protecting human beings from aliens that are disguised as humans. You receive a tip that a Halloween party has been infiltrated by some aliens. Your mission, should you choose to accept (oops wrong movie!), is to identify and capture these aliens in disguise.
In Machine Learning terms, this is an alien identification/classification problem: given a dataset of actual humans and aliens disguised as humans, you want to identify the aliens.
You and your fellow agents head to the party and capture some people who you thought were aliens. You identified some of them correctly and some of them incorrectly! Now, it is time to evaluate your ability to identify aliens disguised as humans using recall and precision.
1. Recall
Of all the aliens disguised as human beings, how many did you identify correctly?
1.1 🥱Boring definition
1.2 👽Interesting (contextual) definition
When you stormed the party and made decisions about who was an alien and who was a human, you identified some aliens correctly and you left some out because you wrongly thought they were humans. Recall is a measure of how good you were at correctly picking out the aliens, out of all the people that were actually aliens. In some sense, recall is also a measure of how good you were at not leaving behind any aliens at the party.
1.3 📈What does high recall mean?
High recall means that there were fewer cases of disguised aliens that you mistakenly thought were humans.
The downside is, that it could also mean that you judged way too many people as disguised aliens. You could identify everyone at the party as an alien and your recall would be the perfect score (you have no false negatives because everyone is considered a ‘positive’ case!). So, there might be a lot of actual humans that you captured who might not be very happy about being interrogated unnecessarily. However, if your priority was to capture as many actual aliens as possible and you do not care as much about mistakenly capturing some actual humans, then recall is the metric for you. Hey, at the end of the day the humans might be pissed off but they are safe!
1.4 📉What does low recall mean?
Conversely, low recall just means that you were not that good at picking out aliens from all the actual aliens. You should probably get more training!
1.5 💵A real world example that makes c̶e̶n̶t̶s̶ sense
In the domain of online transactions, you might want a high recall for a fraud detection scenario. You might wrongly flag some transactions as fraudulent, but with a good recall you are more certain that you have managed to catch a large proportion of fraudulent transactions. Some of your customers might feel a little frustrated for having their transactions being considered fraudulent but your customers/company is less prone to unfair loss of money.
2. Precision
Of all the people that you identified as aliens, how many were actually aliens disguised as humans?
2.1 🥱Boring definition
2.2 👽Interesting (contextual) definition
When you identified and captured some people thinking they were aliens, you captured some aliens and some innocent humans. Precision is a measure of how many people were actually aliens, out of all the people you thought were aliens. In some sense, precision is also a measure of how good you were at not wrongly identifying actual humans as aliens.
2.3 📈What does high precision mean?
High precision means that there were fewer cases of actual humans wrongly identified as aliens.
It could be that you only identified and captured 1 person that you thought was an alien and the person actually turned out to be an alien in disguise. Voila! Numerically, your precision is perfect. The downside is that you might have left a lot of aliens lurking at the party in disguise. However, lets not forget that the MIB is a secret agency and you do not want to wrongly arrest an actual human and jeopardise the covertness of the MIB or the covertness of the fact that aliens exist and lurk among us in disguise! In such a scenario, precision is the right yardstick.
2.4 📉What does low precision mean?
Conversely, low precision means that you might have captured way too many actual humans thinking they were aliens. You have no choice but to erase their memory using a neuralyzer, before they reveal the existence of aliens and the MIB to the world!
2.5 💵A real world example that makes c̶e̶n̶t̶s̶ sense
In the domain of banking, the problem of identifying loan defaulters is one where you would want high precision. If you wrongly identify too many customers as loan defaulters, your bank would not be able to lend money to enough people. The bank’s revenue earned from the interest accrued and paid by the borrowers will shrink and that’s not good for the bank’s bottom line!
3. F1 Score
I care about both precision and recall so I want to strike a balance between the two! — Your boss at MIB
It is now the end of the year. Which also means, it is time for your end-of-year performance appraisal with your boss. In order to disburse the year end bonus fairly, your boss needs to compare the performance of all the MIB agents in terms of the agency’s overall goals. The agency’s goals are twofold — the MIB needs to successfully capture aliens but it also needs to maintain its secrecy and keep the world blissfully unaware of the presence of aliens. Should your boss use precision or recall?
One possible solution is to use the F1 score — it helps to strike a balance between precision and recall.
F1 score is the harmonic mean of precision and recall:
4. Why not keep things plain and simple and just use accuracy?
I hear you. I, too, am a lover of simplicity in Machine Learning but certain problems are such that using accuracy as our metric of measuring the performance of our classifier may not be wise. I am, of course, talking about problems with imbalanced classes i.e. our data is distributed across classes in an imbalanced manner. Let us say the aforementioned Halloween party had a 100 people and only 5 of them were aliens in disguise. In this case, if you identified all of the 100 people as humans you would have the stellar accuracy of 95%, but that is unlikely to fetch you a good bonus at the end of the year, when your fellow agents with good F1 scores help the MIB achieve its core objectives. Hence, considering precision, recall and F1 score are a viable alternative to measure classification performance.
5. Final Takeaways
- When classes are imbalanced, precision, recall and F1 score are a wiser choice than accuracy.
- Use recall when you care more about discovering as many of the true cases as possible.
- Use precision when you care more about being right about the cases that you identified as positive.
Meet the Author:
Hi! I am Aish. The first two letters of my name are “AI” and AI and Machine Learning are what I am passionate about. I am currently a Senior Data Scientist and Machine Learning Solutions Architect in Singapore. You can reach out to me via LinkedIn.
