A common man’s guide to MAE and RMSE
Understand in simple words — What do they mean in the real world and when to choose which?
A businessman, a prospective client of mine, asked me yesterday when I showed him my forecast models, “How accurate do you think these will turn out to be?”. I was ready for the question. “Very”, I said with quite some confidence. “It has an extremely low MAE”.
Hiding his smile at my naive answer, and looking at me the way an elderly looks at a teenager yet to understand the ways of the world, he asked again. “I am not very good at understanding advance financial and mathematical terms, could you explain it to me in some simple way, so that I understand how much risk am I taking here, believing your forecasts?”.
Common sense isn’t that common and simplicity isn’t that simple to attain.
So I politely asked for another day’s time and took his leave.
Mistakes maketh the man
We understand very clearly in our lives that the same magnitude of mistake carries different weights in different situations. Being late by 10 minutes might not be bad when you are attending a casual get-together dinner with friends, but is a life-altering catastrophic event when it is being hosted by your wife’s parents.
You juuust might be able to get away by saying that you are quite punctual, most of the times. You are basically asking to be considered on your average performance.
That pretty much, is Average Error or Mean Error for you.
We should think this in the context of my client. He is the owner of a family-run restaurant — not too big, started a few months back. His ask was to predict the daily customer footfall so that he is able to plan for correct amount of raw food materials to be bought from the wholesale market. Being a new business, he is equally afraid of incurring losses due to excessive food being wasted at the end of the day, as well as having to ask customers to go back because he has nothing more to cook for the day.
Let’s visualize this on a weekly scale. Suppose these are the actual number of customers per day:
I create the models, and forecast something like this:
I then calculate the error or mistakes committed by the model:
Total Error = 0. It’s very easy to see the fallacy in this logic. My mistakes in positive direction get cancelled by those in the negative direction.
I say, a mistake is a mistake. Mathematically, that means taking the absolute value of the mistake.
We just understood what MAE (Mean Absolute Error) is.
It says that on an average, I will mis-predict a day’s customer count by 6.285. Whether I will predict more or less, is not obvious by knowing the MAE. My client might be wasting food of 6.285 people each day or turning away 6.285 customers each day — on an average.
For the sake of understanding, let’s assume that my model makes mistakes only in the positive direction, that is, I might make a mistake of saying, “There would be 25 customers today”, when there actually may be only 20. But never will there be a case where I under-predict the number to be only 15. This basically means food might get wasted, but all customers are entertained.
One’s a Thief, Another’s a Robber
So does that mean two approaches (models) having same MAE are equally good (or bad)?
Imagine a competitor of mine, whom I will refer to as BloodyC Pvt. Ltd., is also trying to sell its approach to this gentleman who owns the restaurant. Our predictions look something like this:
I make small mistakes every day, while BloodyC is accurate on most days but commits bigger mistakes on some days. Which is better?
The answer can not be derived by looking at numbers alone. It’s about the business, its vision, its morals and the people behind it. So I will ask you to imagine a situation, not far removed from reality, where my model comes out better.
(Data) Science is never about numbers and logic. It’s always about emotions and lives.
In many countries, shops and homes, when there’s excess food left, we try to give it to those who need it more than others, or feed pets & stray animals if it’s not harmful to them. Everyone has limits, of course. So if there’s food left only for 2–3 people it’s easier to immediately find someone needy but a bulk is often thrown out into the garbage bin if the quantity’s more. Not every shop owner connects to a homeless shelter or similar service.
My prospective client, the restaurant owner, behaves similarly. It’s also an emotional pain for many to see so much food go to waste. Although it’s the same financially to him, he would much rather prefer my model, which wastes little each day as compared to BloodyC’s which on a couple of days, wastes a lot.
Find the situation too abstract or hypothetical? Here’s a very real one:
$crooge McDuck
Professional traders often prefer investing/trading in the stock market via derivatives — futures & options. In many strategies, bets are made on the price range within which a stock or derivative will stay for the next n number of days.
Does my model now feel a safer bet than BloodyC’s? Even when I go wrong, I go wrong by a small margin. A trader will still make money in that limited range, even if his bet was not accurate. If the price had swayed away a lot from the prediction on which the trader had trusted, it means losing a huge amount of money in one bad investment — trusting BloodyC would have cost him dearly.
If this makes you feel that my model is much more suitable for your business case, then which metric do you use to express this while comparing the two models? MAE certainly is unable to penalize one more than the other.
Enter, RMSE
RMSE is square Root of Mean Squared Error. So if you square each mistake made in the prediction, and add them up, then divide by 7 (total number of predictions made), you get MSE. If you want RMSE, just do an additional square root. (Phew, wasn’t that a mouthful!)
Let’s see how RMSE looks for our predictions:
Bigger mistakes, bigger penalty — that’s one of the features of RMSE. My prediction got a lower RMSE (2.138) value than BloodyC’s (3.505), which means it is better when evaluated with this objective.
Pray to RMSE!
Does this mean we should always use RMSE to check a model’s accuracy?Absolutely not!
If instead of rice, chicken and Indian curry, my prospective client sold pickles which comfortably last for years, allowing one day’s excess to easily be sold the next day, then he would more likely be benefited by focusing on MAE or any other similar metric.
There are several other nuances of RMSE, MAE, MAPE (Mean Absolute Percentage Error) and one should certainly consider reading about those and diving deeper into the maths behind it. Few articles I found interesting were this and this (jump to Section 11: Accuracy Metrics for Time Series Forecast), for those who wish to explore further.
Parting Thoughts
I hope my prospective client would now be able to understand how much, and what type of risk he is taking by trusting my forecasts. His questions certainly made me revisit what fits his use case better.
As for the not-so-punctual husband, he can only hope not to be judged on RMSE when he goofs-up big in the near future!
Interested in sharing ideas, asking questions or simply discussing thoughts? Connect with me on LinkedIn, YouTube, GitHub or through my website: I am Just a Student.
See you around & happy learning!
