The world’s leading publication for data science, AI, and ML professionals.

Understanding Correlation And Diversification

Why It Pays To Invest In Uncorrelated Assets

Correlation is a fundamental concept in both finance and statistics. In plain English, correlation tells us the likelihood that two variables move together. A high correlation means that when one variable goes up, the other is very likely to go up as well. The stock returns of similar companies, like Coke and Pepsi, are positively correlated:

A negative correlation means the opposite (when one variable goes up, the other variable usually goes down). Stocks and Treasury bonds tend to be negatively correlated.

A correlation of zero equates to statistical independence. If two variables are statistically independent, it means that each has no bearing on the other. If you and your best friend each throw a six sided die, the results will be independent (no matter how much of a mind meld you two might share).


Variance and Uncertainty

Time for a little bit of statistics. The variance of a variable is a measure of how much it well… varies. You can picture variance by imagining a game where you drop a feather on the ground and attempt to predict where it will land. If there’s no wind that day (or you are playing indoors), then it’s pretty easy. Without a gusty wind to blow it around randomly, we know that the feather can’t go too far. Thus, we can say that the final potential landing spot of the feather has a low variance:

If you happen to drop your feather in the middle of a tornado, then it’s anyone’s guess where the feather will end up (it could land several miles away or more). The extremely large range of potential landing spots is an example of high variance:

Another way to think about it is that low variance means low uncertainty and high variance means high uncertainty. So it makes sense that where possible in finance, Data Science, and life in general, we want to decrease variance. Less variance means outcomes are more predictable, plans are less likely to go awry, and we are less likely to take a loss (whether it be a financial or emotional one). Therefore, we can spend less time and resources on insurance, contingency plans, and hedges.


Low Correlations Diversify

That’s where correlation comes in (or the lack thereof). Let’s say we have two normally distributed random variables, X and Y. Both have a mean of 0, a standard deviation of 1, and a correlation of 0 against each other. For a primer on the normal distribution, check out this blog post.

What’s the standard deviation of the average of our two random variables? We need to introduce a few mathematical rules to figure this out:

  • Variance is the square of standard deviation (so in our example both X and Y have a variance of 1, since 1² = 1):

Var(X) = Stdev(X)²

  • When we multiply a normally distributed random variable, X, by a constant W, the variance scales by the square of the constant:

Var(WX) = W²Var(X)

  • The variance of the sum of two random variables, X and Y, is equal to the sum of their variances if X and Y are independent:

Var(X+Y) = Var(X) + Var(Y)

Taking the average is like applying a constant of 0.5 to each random variable, and then summing. So we can combine our two rules above to figure out the variance of the average like so:

Var(0.5X + 0.5Y) = 0.5² Var(X) + 0.5² Var(Y)

Cool, now we can calculate the variance and the standard deviation of the average of X and Y (let’s use the actual numbers – Var(X)=1 and Var(Y)=1):

Var(0.5X + 0.5Y) = 0.5² 1 + 0.5² 1 = 0.25 + 0.25 = 0.5

To simplify our notation, let’s refer to the average of X and Y as A. We can calculate the standard deviation of A easily because we already know its variance:

Stdev(A) = Sqrt(Var(A)) = Sqrt(0.5) = 0.71

Wait what happened? X and Y both have standard deviations of 1 and yet their average has a standard deviation of 0.71 – the average of the variables is less volatile than the individual variables themselves. I want to emphasize two keys here:

  1. Taking an average is a lot like creating a portfolio. Here, we took the average of X and Y by multiplying each by 0.5. But we could easily call X Apple stock returns and Y Google stock returns. Then we would interpret the 0.5 as a weight. And the average of X and Y becomes the return of a portfolio that’s split evenly between Apple and Google stock. So when we make a portfolio of investments, we are effectively taking a weighted average of their individual returns.
  2. The variance (and standard deviation) of the average of X and Y is lower than their individual variances. This occurs because they move independently of each other, which diversifies (one zigs while the other zags, so that the average movements of the group is less volatile than any one individual’s movements). Another way to understand this is The Law of Large Numbers. The more independent observations you take, the more likely that the average of those observations will be close to the true value (so the average of 100 observations would have a lower variance than the average of a mere 3 observations).

It’s important to note that this effect only occurs when the variables have correlations less than 1 (ideally a lot less than 1). If two things are perfectly correlated, then we can’t expect their movements to diversify – rather, they would move together in perfect lockstep (and even if we averaged thousands of these perfectly correlated variables, the variance would not decrease).

Also bear in mind that the average of two uncorrelated variables will not always have a lower variance than the individual variances of the variables. When one variable has a high variance and the other a low one, the variance of the average of the two variables will be between the two individual variances. But because of diversification, the actual variance of the average will be significantly less than the arithmetic mean of the the individual variables’ variances.

We saw how the standard deviation declined from 1 to 0.71 when we averaged two independent variables. What if we have access to more uncorrelated variables to add to our portfolio? The following plot shows what happens to standard deviation as we incrementally add more uncorrelated random variables:

The first few variables added cause standard deviation to drastically decline before diminishing returns set in. This is expected – the more variables that we’ve already included in our average, the less we should expect each new variable added to impact things. But even so, we can see the Law of Large Numbers in action – the more variables we average, the lower the variance, and the more certain we can be. Switching to Finance for a second, the more uncorrelated investments we can find, the lower the variance (and standard deviation) of our portfolio’s return, and the more certain we can be that we won’t lose money. To see why the finance industry uses the portfolio’s standard deviation as a proxy for its risk, check out my previous blog.

This is exactly what would-be world beating hedge funds attempt to do. They try to find as many uncorrelated return streams as possible, combine them in a portfolio, and then hope to print money. Often it doesn’t work out quite as planned, but it’s a nice dream.


Practical Considerations

Ok, now that we are all jazzed to find as many uncorrelated investments to fill our portfolios with, it’s time for me to rain on our parade a bit:

  • Truly uncorrelated investments are really hard to find. And you want your investments to not only be uncorrelated now but also uncorrelated in an economic crisis (when you really need the benefits of diversification) – and those are even harder to find. In reality, most investments will be at least somewhat correlated, which means that combining them will still reduce variance, just not as much. Mathematically speaking, the lower the correlation between two investments, the more they diversify (see here for the equations). The following plot shows how changing the correlation between two normally distributed random variables changes the standard deviation of their average:

  • The classic risky investment that almost everyone owns is stocks. So it would be great to find an investment that is negatively correlated to stocks, allowing us to hedge away their risk (and reduce the variance of our overall portfolio). Unfortunately, everyone wants this. So any investment that exhibits a negative correlation to stock returns such as Treasury bonds, put options, or VIX futures gets priced at a premium and will probably earn you low or negative returns (hey, insurance is never free). Treasury bonds are interesting because over the past few decades their returns have been negative correlated to that of stocks, and they have a positive expected return (unlike other hedges that you must outright pay for). Thus, a core allocation to stocks and bonds is the lynchpin of most long term portfolios. However, the negative correlation is not guaranteed – there have been multi-year periods (like the 1970s) where stock and bond returns have been highly correlated (a sustained increase in interest rates would usually hurt both stocks and bonds).
  • Which brings me to my final point – we want a positive portfolio return (we want the average of our random variables to be greater than zero as frequently as possible). This means that uncorrelated random variables with zero or negative expected returns are useless to us. Yes, adding them to our portfolio would reduce the overall variance, but it would also reduce the expected return (the weighted average), which would be counterproductive.

This post is Part 3 of my primer on investment portfolio optimization. If you haven’t already, check out Parts 1 and 2 as well:

Understanding the Normal Distribution

Understanding Investment Risk


Related Articles