GuidePost

The GRABot BAG
by Sherrie E. Grabot, CEO

May 2009
Is Monte Carlo Simulation Too Much of a Gamble?

A recent article in the Wall Street Journal suggests that one of the most widely used tools for modeling investment results, the venerable Monte Carlo simulation, is in need of significant repairs. The argument is that most of the modeling methodologies based on this technique fail to account for large market swings, particularly one as massive as the one we're now experiencing. Retirement simulations painted an overly-optimistic picture of the future, leading investors to be overly confident in their expectations.

We find it very ironic that during more bullish times, GuidedChoice was actually criticized as being too pessimistic in comparison to other advice providers. Because of the assumptions used, our simulation projections never showed possible outcomes below a 50% probability. That way no one would be tempted to 'bet' on an unrealistic 5% chance of reaching their goal because they were lured by the bigger pot at the end. Since our participants were never shown optimistic numbers, the only way to reach their goals in many cases was to increase their savings rate. Setting an appropriate savings rate is by far the most important focus of our GuidedSavings solution, and a recommendation to save more is the result the system most often delivers.

Another irony here is that although our co-founder Dr. Harry Markowitz won his Nobel Prize for modeling asset allocation, he would say unequivocally (and often does) that even the best asset allocation strategy can't get you an adequate retirement income if you don't save enough along the way.

The uncertainties of risk
Fast forward from the boom times to the gloom of today. Now many in our industry are being criticized for being too optimistic. Optimism is closely tied to risk tolerance; one might say an appetite for risk is just a rather specific type of optimism.

Helping individuals assess their risk tolerance is another key focus of GuidedSavings. Once risk tolerance is determined, then the optimization portion of the process (which translates into asset allocation) is easy. So the essential question is: can an individual using the tool appropriately set their risk tolerance, or does this assessment tend to be unrealistic? When viewing possible future scenarios, people tend to focus only on the up side, and not the down.

As our Investment Committee has always maintained, and as recent events have abundantly proved, individuals often have a lower risk tolerance than they, or the experts, think they do. It's well known people feel the pain of a given loss far more than the joy of an equivalent gain. But in investing, the reality of loss aversion may be deferred for many years after the initial decision about risk. When the loss does come and the reality sets in, it is far too late.

Short-term distortions
To combat this tendency, GuidedSavings incorporates methodology to ensure a realistic long-term perspective. That does not mean we can predict the outcome for any asset class or particular investment in the short run. In an extreme down-market environment, which is generally short-term, all the correlation coefficients will move toward 1.0. In that scenario, the conclusion drawn utilizing only short-term data could be that diversification provides no value. Historically and intuitively we know that to be untrue -- so obviously one cannot use short-term data to form accurate long-term conclusions.

We cannot claim perfection for our methodology, or anyone else's. We can, however, assert that these are the best tools currently available. They will continue to improve as new information and approaches are added. And they certainly provide a better result than guessing -- which unfortunately is the default strategy for many retirement investors.

Specific concerns
With all that being said, the WSJ piece raises some important points. These deserve a few specific answers.

1) Risk cannot be accurately measured.

In fact, long-term risk has been measured relatively accurately for years. Short-term risk has also been measured fairly well. However, we feel it is essential to communicate to GuidedSavings users that the short-term risk of any investment, even an explicitly guaranteed one, has the potential for a 100% loss. Therefore, we never predict 100% probability for any investment mix; the highest we show is 98%. The bottom line is that there are no guarantees -- and this is as true for Social Security and private pensions as it is for 401(k).

Catastrophic economic events, such as what we are experiencing, have hit all retirement systems equally hard. We just see the tangible effect more clearly on our 401(k) statements -- which we would argue is a good thing for people. We may not see the effects on our social security or pension system for years to come, but ultimately those effects are just as real. Those systems just allow the average person to maintain a false sense of security during the crisis itself.

2) The problem isn't the Monte Carlo simulation, but the underlying assumptions, especially if end users can modify those assumptions.

It would indeed create problems if individuals could alter basic, long-term assumptions such as inflation and market volatility. However, GuidedSavings does not allow users to modify the assumptions that go into our simulations, so we avoid that issue entirely. In addition, the assumptions set by our Investment Committee are relatively conservative ones -- the kind of pessimistic approach that critics are calling for.

3) The ideal model runs tens or hundreds of thousands of scenarios.

This is the kind of statistical rigor that computational methods make possible, and it is exactly what we do. Depending on the tool and process being analyzed, for each case we run many simulations monthly over long periods of time. For GuidedSavings, we may be running hundreds of random draws each month. Over a 40-year example horizon, that adds up to at least 240,000 simulations.

4) The bell-shaped curve does not weight unusual events appropriately.

To begin with, the distribution of investment results that we see annually is not bell-shaped at all (the classic "normal" or "log-normal" distribution pattern). But according to the law of large numbers, even if the true probability distribution over a given time period is not bell-shaped, or the simulation method is not based on bell-shaped distributions, when one accumulates outcomes over a long enough time the cumulative returns will approach the bell shape.

What exactly does that mean? The shape of the curve is indeed critical, and has been the subject of much academic scrutiny. But whether the "correct" shape to account for unusual, severe market events is bell-like or otherwise, no one really knows.

Finally, a word about expectations. We have always emphasized the lower spectrum of our distribution curve of random draws to ensure a more conservative view (the one our critics called 'pessimistic'). As stated, we only show our users probabilities greater than 50%. So our most "highly likely" scenario is a 95% chance that a participant would have at least the given amount of savings by retirement.

As a result of our approach to communication, the design of our user interface, and the methodology driving our system, fewer than 3% of our GuidedSavings users are invested in all-equity portfolios. The vast majority are at more moderate risk levels, and did not suffer anywhere near the 54% losses cited by the WSJ.

The user, not the tool
If the discussion above points to any conclusion, it might be that any tool is only as good or bad as the craftsman using it. Monte Carlo is a powerful instrument. But if the assumptions it depends on are too rosy, the results will be overly optimistic. If it's used to predict the behavior of unproven assets with poorly understood fundamentals, such as mortgage-backed securities or credit-default swaps, the results will also be unreliable. And when the users are themselves overly optimistic, (as is almost inevitable during a long boom), even the most rigorous methods can skew toward optimism.

No matter how sophisticated our technologies become, we would do well to remember the golden rule from the earliest days of computing: "garbage in, garbage out."

~~ Sherrie