The world’s leading publication for data science, AI, and ML professionals.

A false sense of security

Or why you should be wary when investment firms tell you they ran 1000 simulations

So, a couple of days ago, I got into a little squabble with my dad about what to do with some cash that I had lying around in my bank. I wanted to invest the money to generate a better interest rate than the pathetic ~0.5% offered by most banks. My dad leaned more toward just parking my money with a random advisor and trusting them to invest it while I was more inclined toward managing the money on my own. Unable to come to a conclusion, we decided that an experiment was in an order and decided to split the money in a 2:1 ratio, parking 2/3 of the money with this new, popular investment firm and leaving 1/3 of the money for me to invest with.

However, when I started to look into the firm, I got a little wary – especially since they kept promoting themselves as a "data-driven" firm and had all these cool graphics. Here’s the article outline (and tl;dr)

tl;dr

  1. The firm: Singapore finance company, boasting data driven solutions and proprietary systems
  2. Problems: a. Inaccuracy/Unreliability of simulations: The accuracy of simulations/models are contingent on the accuracy of their input. Running a model 1,000 or 1,000,000 times does not increase its forecasting accuracy if the inputs are flawed to begin with. Hence, their 1,000 simulation forecasts aren’t that impressive. b. Historical Returns + Fees: Historical returns (which is used by the company for projection) are never indicative or provide certainty about future returns. Also, fees affect return rates.

  3. Conclusion: Nothing wrong with investment companies doing such projections but we, as consumers, must be mindful that they aren’t always the most accurate thing

The Company

The firm bills itself as a "Singapore-based financial technology company that empowers people to take control of their financial future". I’ve highlighted the "financial technology company" part because that’s where this company really sells themselves highly, contrasting this ‘advanced’ technological approach to traditional financial advisors. Their mission statement goes on to explain that they use a "proprietary system [to] provide data-driven wealth advice in constructing personalised solutions." Wow – to the ordinary folk, this all sounds really cool and (to use a Singapore colloquialism) cheem. Data-driven advice? Proprietary system? Sign me up BABY.

When one first opens an account with them, you are given the option to set some financial goals. You put in your initial investment, your monthly contributions, and your risk tolerance (represented by maximum % drawdown). Here’s where the impressive, data-magic happens. After running some INTENSE calculations, it spits out a projected portfolio growth graph (based off of 1000 simulations; yes ONE THOUSAND). Here’s how it looks:

Pretty fancy no? So, this supposedly data-driven approach appears to convey to potential investors that- Hey! We’re using some cutting edge technology to manage your money and hence you can trust us with it. But here’s the thing… is it really that impressive? Let’s find out.

Problems

Inaccuracy of simulations

First off, running a 1000 simulations sounds really cool. I mean, if we tried it a 1000 times, surely it has to be accurate right? But let’s strip that back a bit and think about what this means. Running 1000 simulations to get a range of possibilities could be considered a Monte Carlo Simulation. This is defined as any "model used to predict the probability of different outcomes when the intervention of random variables is present". Essentially, it allows you to use repeated sampling to account for underlying randomness.

Is it really that ‘advanced’ of a concept? Sure, it sounds and looks impressive but it’s actually easily replicable. So, I created a function to estimate returns, given expected returns and standard deviation (this, as we will discuss later, is also another key issue with the simulation)

After creating the function, I ran a loop on it, allowing users to input their initial investment, yearly rate of return, and standard deviation (as per the firm’s simulation).

For my simulation, I used the yearly rate of return and standard deviation for their 60/40 portfolio – a conservative but growth orientated strategy.

how much is your initial investment? 10000 
What is the yearly rate of return? 0.0771 
What is the standard deviation of return 0.0870 
How many years are you Investing for? 30 
How many times to simulate? 1000

After putting in my settings (following the given rate of returns), I then calculated the 10th percentile, the median, and the 75th percentile of the portfolio (mimicking the results given by the firm’s simulation)

The median return of the portfolio is 71657.24. 
The 10th percentile is 38651.81 and the 75th percentile is 96055.78. The highest amount is 438889.69, and the lowest is 16925.8.

Comparing my results to theirs’ (pictured below), I must say that it was pretty close. My simulation overshot the median by ~$7000, but this was likely because I didn’t take into account fees and stuff. The top 25% also overshot by ~$6000, but the bottom 10% was nicely similar with only a ~$2000 difference. All in all, quite satisfied with the result.

At this point, you might argue, "Zach! See? It’s so accurate! What’s not to like about the firm?" Well, that brings me to the issue with models. The output of models depends entirely on the INPUT of the model. This may seem obvious but that’s exactly the problem with these fancy-shamancy things that tell you they ran 1000 simulations. I can run 1,000,000 simulations (I did in fact do that LOL) and it wouldn’t increase my accuracy one bit if my input was completely wrong. For instance, here’s me running the model again but with different inputs (and simulating it 10,000 times).

Did me running the model 10,000 times make it any more accurate? NO! If my input is flawed (in this case, a 70% annual return), my output will still be flawed, regardless of how many simulations I ran. So, these things look nice and fancy but they cannot be markers of "accuracy" because they are heavily reliant on the input. Which brings me to my next point…

Historical Returns + Fees

The firm, as a company, has only been around since the late 2010s and hence the rate of returns projected comes from the historical returns of the funds that they invest in. The projected rate of returns, when graphed, look like this:

With a standard deviation of 8.7%, what this means is that there is a 68% chance (one standard deviation), that your yearly rate of returns will be between 6.9% and 8.4%. The firm, as a principle, does not ‘stock pick’ but instead helps clients to diversify by buying into equity and bond ETFs/mutual funds (and charging a management fee for intelligent rebalancing). The projections are hence not of the firm’s portfolio’s historical return, but an amalgamation (with weighting) of the historical returns of these funds! And we all know the issue with using historical returns… As David Blanchett (head of retirement research for Morningstar Investment Management) notes, using historical returns to gauge future returns "implicitly incorporates the recurrence of historical events and allows for only a limited amount of data". Furthermore, he argues that "a problem with Monte Carlo tools that is exacerbated today is that they can often paint an unrealistic picture of returns", giving a semblance of certainty when there is, in fact, "considerably less certain[y]".

Another potential factor that will affect returns are fees. I understand that every investment has to charge fees. After all, someone has to be paid to keep the company running. The firm compares their 0.6% fee (for below $200,000 portfolios) to the industry average of about ~2%. While this is surely lower, we have to remember once again that the firm really isn’t an investment company per se. Rather, they allocate money into funds and the fund managers are the ones that invest the money. These funds also charge management fees (which add up to ~0.5%).

Conclusion

So, where does all this leave us? My intent in writing this was not to discredit the firm or discourage anyone from investing with such ‘data-driven’ investment companies. Rather, by dissecting the ‘fanciful’ technology behind the company, I wanted to bring out two key points.

  1. First, be wary of nice visuals and cheem sounding things. Running a 1000 simulations sounds really cool but you have to remember that the forecasting output of a model is only as accurate as its input. If the output is inaccurate, a 1000 or a million simulations really won’t make that much of a difference to the model’s accuracy.
  2. Second, historical returns do not provide certainty for future returns. We can definitely use historical returns to approximate what future returns will look like BUT we need to remember that there is certainly no certainty, no matter how nice the graphs look.

So, I’m not knocking them. The firm certainly appears to charge lower fees and does appear to have sound strategies of diversification etc. BUT, as a potential investor, just be aware that there are limitations to the beautiful data-driven models that they present as their ‘edge’.


Originally published at https://zachlim98.github.io/me/.


Related Articles