The world’s leading publication for data science, AI, and ML professionals.

What is the Bias-Variance Tradeoff?

Explaining the Differences Between Bias and Variance

Data Science Concepts

What exactly is the Bias-Variance Tradeoff and why is it important?

How they differ from each other

Photo by Alex on Unsplash
Photo by Alex on Unsplash

When it comes to Data Science, there comes a point where one has to make sure that their Machine Learning model is "just right". There are two different sides to every machine learning model: Bias and Variance. As data scientists, we must search for a model that strikes the perfect or at least close to perfect balance between the two. Like Goldilocks in Goldilocks and the Three Bears, we must search for a model that is not "too hot" or not "too cold".

This is the overall concept of the "Bias-Variance Tradeoff". Bias and Variance are errors in the machine learning model. As we construct and train our machine learning model, we aim to reduce the errors as much as possible. In an ideal situation, we would be able to reduce both Bias and Variance in a model to zero. However, if Bias were to decrease to zero, then Variance will increase, and vice-versa. So in order to optimize our model, we must be able to find a balance between the two where we can decrease both Bias and Variance.

Sign up for a Medium Membership here to gain unlimited access and support content like mine! With your support I earn a small portion of the membership fee. Thanks!


What is Bias?

Bias is an error in the Machine Learning model. That error being the difference between the model’s predicted values and the actual values.

In statistics, Bias is the tendency for underestimation or overestimation of a parameter. Bias can come in many different forms such as Sampling Bias or Selection Bias.

Photo by Romson Preechawit on Unsplash
Photo by Romson Preechawit on Unsplash

What is Variance?

Variance is also an error in the machine learning model. The error this time is the model’s sensitivity to the data. If it is too sensitive then the model may see signals and patterns that are not actually there.

Statistically, Variance is the distance between each variable in the dataset and the average or mean value of all the variables. It measures the spread of the dataset and how far each variable is from each other.


Bias vs. Variance

Between Bias and Variance, which error should be reduced the most? And what happens if you were to choose to reduce one error over the other? An error is an error so overall it is not better to have one or the other.

Underfitting & Overfitting

If we were to prioritize the reduction of one error over the other, then we would end up with a model that either overfits or underfits the data. This means that the model will not provide usable results for any new data. In the case of underfitting, the model is unable to fit to the training data and as a result, will be unable to generalize new data. Overfitting will result in a model that models the training data too well; detecting patterns and noise in the data that are not actually there. This can negatively impact the results from the modeling of any new data.

Reduce Both

Because of the effects of prioritizing the reduction of one error over the other, we must then reduce both. This will lead us to the best result we can hope for.

Accuracy in Bias-Variance Tradeoff
Accuracy in Bias-Variance Tradeoff

Take a look at the image of the four targets and the effect Bias and Variance have on the accuracy. As Bias increases, the results land off the mark completely albeit consistently. As Variance increases, the results sometimes land near the mark but are much too scattered. If both the Bias and Variance were high, then the results are scattered and completely off the mark as seen in the target in the top right. The best option would be the target with the lowest Bias and lowest Variance, which has the results land consistently on the mark.

In order to achieve an optimum model, Bias and Variance must both be lowered as much as possible.


The Optimum Model

Lowering both Bias and Variance means reducing the total error in the model. It also means creating a model that is not too simple and not too complex. Model complexity is something that we also look at when considering the Bias and Variance. A model that is too complex is a model that is overfitting and a model that is too simple is underfitting. This also correlates with the direction of the Bias and Variance.

Looking at the image of Error and Model Complexity and we can see the point where an optimum model is achieved.

Plotting the Bias-Variance Tradeoff
Plotting the Bias-Variance Tradeoff

Imagine the image above like it is a large valley. Even though it might be more admirable to reach the peak of a mountain, in the case of the Bias-Variance Tradeoff, we want to arrive at the bottom of the valley. The bottom or lowest point of the valley is where we want the Bias, Variance, and Total Error to remain as we create our machine learning model.

Once we are able to realize the lowest possible Bias and Variance, then we have achieved the Optimum Machine Learning Model.


Closing Thoughts

As you have been able to see throughout, when it comes to Bias and Variance, reducing one over the other is not the preferred method. Bias is an error between the actual values and the model’s predicted values. Variance is also an error but from the model’s sensitivity to the training data.

If we were to aim to reduce only one of the two then the other will increase. A prioritization of Bias over Variance will lead to a model that overfits the data. Prioritizing Variance will have a model underfit the data. Neither of these outcomes are favored. However, there is a point where we can reduce both Bias and Variance without affecting the other and that point is what we are searching for.

In order to achieve the best machine learning model, we have to balance out the reduction of both. This will give us a model that is not too simple (underfitting of the data) and not too complex (overfitting of the data). A balance in the reduction of the two errors will lead us to the optimum model complexity. This is the overall concept of the Bias-Variance Tradeoff.


_Follow me on Twitter: @_MarcoSantos_


Related Articles