MSE, RMSE, MAE, MAPE, sMAPE… to name just a few. There is a vast ocean of different error metrics out there, each one with its set of pros and cons and supposedly covering more cases than the previous ones. So how to decide which metric to use for our projects?
I believe that the key to answering this question is knowing the strengths and limitations of the most popular metrics. This way, we can choose the metric most suitable for the task at hand. That is why in this article I cover two of the metrics I have recently worked with.
Mean Absolute Percentage Error (MAPE)
The mean absolute percentage error is one of the most popular metrics for evaluating the Forecasting performance. It is given by the following formula.
Where _At stands for the actual value, while _Ft is the forecast. In this case, we can interpret t as either observation in case we are doing a generic regression problem (predicting the weight of a person or the price of a house) or as the time index in the case of time series analysis.
The formula often includes multiplying the value by 100%, to express the number as a percentage.
Advantages
- Expressed as a percentage, which is scale-independent and can be used for comparing forecasts on different scales. We should remember though that the values of MAPE may exceed 100%.
- Easy to explain to stakeholders.
Shortcomings
- MAPE takes undefined values when there are zero values for the actuals, which can happen in, for example, demand forecasting. Additionally, it takes extreme values when the actuals are very close to zero.
- MAPE is asymmetric and it puts a heavier penalty on negative errors (when forecasts are higher than actuals) than on positive errors. This is caused by the fact that the percentage error cannot exceed 100% for forecasts that are too low. While there is no upper limit for the forecasts which are too high. As a result, MAPE will favor models that under-forecast rather than over-forecast.
- MAPE assumes that the unit of measurement of the variable has a meaningful zero value. So while forecasting demand and using MAPE makes sense, it does not when forecasting temperature expressed on the Celsius scale (and not only that one), as the temperature has an arbitrary zero point.
- MAPE is not everywhere differentiable, which can result in problems while using it as the optimization criterion.
For more information on using the MAPE in a business setting, please see this article.
symmetric Mean Absolute Percentage Error (sMAPE)
Having discussed the MAPE, we also take a look at one of the suggested alternatives to it – the symmetric MAPE. It was supposed to overcome the asymmetry mentioned above – the boundlessness of the forecasts that are higher than the actuals.
There are a few different versions of sMAPE out there. Another popular and commonly accepted one adds absolute values to both terms in the denominator to account for the sMAPE being undefined when both the actual value and the forecast are equal to 0.
Advantages
- Expressed as a percentage.
- Fixes the shortcoming of the original MAPE – it has both the lower (0%) and the upper (200%) bounds.
Shortcomings
- Unstable when both the true value and the forecast are very close to zero. When it happens, we will deal with division by a number very close to zero.
- sMAPE can take negative values, so the interpretation of an "absolute percentage error" can be misleading.
- The range of 0% to 200% is not that intuitive to interpret, hence often the division by the 2 in the denominator of the sMAPE formula is omitted.
- Whenever the actual value or the forecast has the value is 0, sMAPE will automatically hit the upper boundary value.
- Same assumptions as the MAPE regarding the meaningful zero value.
- While fixing the asymmetry of boundlessness, sMAPE introduces another kind of delicate asymmetry caused by the denominator of the formula. Imagine two cases. In the first one, we have A = 100 and F = 120. The sMAPE is 18.2%. Now a very similar case, in which we have A = 100 and F = 80. Here we come out with the sMAPE of 22.2%.
Conclusions
In this article, I described two popular measures of performance evaluation for regression problems. While it fixes some of the shortcomings of MAPE, there are still a few issues left and some new ones created on top. Some other metrics you could investigate are:
- Mean Absolute Scaled Error (MASE)
- Mean Directional Accuracy (MDA)
- the logarithm of the accuracy ratio (the ratio of the forecasted to the actual value)
As always, any constructive feedback is welcome. You can reach out to me on Twitter or in the comments.
In case you found this article interesting, you might also like:
My 10 favorite resources for learning data science online
5 types of plots that will help you with time series analysis
