Laurel & Hardy in Machine Learning
“Well, here’s another nice mess you’ve gotten me into!”
This week I go even further back in time with my references, all the way back to the 1930s. In case some of the Millennial readers just thought “OK Boomer”, I’d like to clarify that I am a Millennial myself. It’s just that sometimes old-timey references make so much more sense than modern ones.
So for those of who don’t know of Laurel & Hardy, I urge you to go onto Youtube and watch a few videos before reading on. For those of you who have been fortunate enough to experience this duo in action, I hope you will see the parallel with what I am about to say.
The Laurel & Hardy of Machine Learning are known as Bias and Variance. Like Laurel & Hardy, Bias and Variance are joined at the hip. While both can lead to significant errors in your model, unlike Laurel and Hardy, who understood the power of errors in making people laugh, in Machine Learning errors are no laughing matter. If you’re not careful with Bias and Variance, they can downright cost you your job.
To help you avoid getting into a nice mess, let’s see why the Laurel & Hardy of Machine Learning deserve your attention.
Variance
“You can lead a horse to water but a pencil must be led.” — Laurel
Fans of the show will remember that Laurel had genuinely good ideas. He would come up with an interesting way of doing something, and share it with Hardy, who in turn would go “Tell me that again”. As soon as those words were uttered, Laurel would trip up and give a completely nonsensical version of his idea the second time around.
Laurel exemplifies Variance in Machine Learning. Variance is your model’s error when it is not able to generalize to data it hasn’t seen before. If you change the data even slightly, the model’s predictions are completely off. Like every iteration of a “Tell me that again” led to a different answer by Laurel, the same happens with your Machine Learning model if it has very high variance.
In other words, your model will learn the noise along with the signal so it simply won’t be able to separate things if there is a missing from lead!
Bias
“ You’re actually using your brain. That’s what comes from associating with me.” — Hardy
Hardy, on the other hand, never understood Laurel when he first shared his ideas, which, at least to the viewer, were clear. However, Hardy had an uncanny ability to understood the non-nonsensical version that Laurel blurted out the second time around.
Clearly Hardy had a lot of difficulty separating the signal from the noise and that’s what Bias is all about. In a simplified sense, Bias can be defined as an error that is caused by your model not being able to learn anything from the training data.
In other words, you’re going to look smarter if you simply use your brain than use a model that has high bias.
So what?
Here comes the usual so what? Why should you care about these terribly forced analogies? You have to care about them because every Machine Learning problem is about balancing the trade-off between the two.
If your model has very high bias, it simply isn’t learning enough (or even learning at all), so that beats the point of Machine Learning. This might happen, for example, because you’re using a Linear Regression for a non-linear problem. So, one way to go from being a Hardy to becoming a Laurel is to use more complex models.
If you then make your model too complex, like using a convoluted backpropogating SVM, Laurel starts hallucinating pencils and horses like I just hallucinated the model’s name, so you want to avoid that too.
Basically, what you need to do is find a trade-off between the two, a happy place where Laurel & Hardy and you live in perfect harmony.
Until you get yourself into another nice mess that is! but that’s what the life of a Data Scientist is all about, isn’t it?
Now if there are any of you that have suffered this far, below I provide a link for a better explanation of the bias and variance trade-off, one that might help your brain recover from what you just read:
