Do Monte Carlo simulations say anything about system robustness?

This past week I received an email where a regular blog reader asked me if Monte Carlo simulations could be used to determine whether a system is more or less likely to fail in the future (how robust the system is) as there seem to be several software packages that offer Monte Carlo simulations as a way to measure strategy robustness. Today I want to talk about Monte Carlo simulations, the types there are, what they do tell us and what they cannot tell us about trading strategies. I will also talk about robustness, what this actually means and whether or not Monte Carlo simulations can or cannot help us discriminate strategies according to their level of robustness. In the end you’ll see that Monte Carlo simulations are no magic bullet and offer only limited insights into possible trading system robustness.

Selection_999(055)

First of all let us talk about what Monte Carlo simulations are. In general these simulations randomize some measure of trade returns in order to obtain alternative equity curves that share the same distribution of returns. The idea is that a system that behaves as expected will take the same types of trades in the future, only that the order of these trades will be randomized as market conditions change. The strategy will remain exactly the same at its core, the distribution of returns will be the same, but the equity curves will vary significantly depending on each random draw. In general Monte Carlo simulations are limited by some fundamental assumptions – they do not work for every strategy type – and they give some important information related to worst case boundaries for strategies as we are able to tell whether a strategy’s results are likely to be draws from the back-testing distribution of returns at a given confidence interval.

But does this have anything to do with robustness? Robustness is the ability of a system to work, to survive to future market conditions. Systems that are more robust are systems that belong to a group which has a smaller tendency to fail under new market conditions than another system group. For example strategies developed over 10 years of data are more robust than strategies that are developed over 1 year because the first group has a much larger probability to survive a year of unknown market conditions. This can be shown to be true for at least the past 30 years across many different Forex symbols. But do Monte Carlo simulations can in any way improve robustness? In reality, they do not.

Selection_999(056)

Monte Carlo simulations describe the boundaries of trading system behavior, these boundaries have little to do with future probabilities of survival. If a trading system can be said to fail by Monte Carlo simulations if it has faced a 5% drawdown over 1 month it is not more or less robust than a trading system that would fail at any different drawdown level within this same period. These different systems would only show different levels of risk – different amount of losses needed before discarding – but their probability of future failure is unrelated to this. A Monte Carlo simulation with better overall distributions or statistics only points to a better system under in-sample conditions – a system with a better distribution of returns – but it tells nothing about future survival. The boundaries determined by Monte Carlo simulations are boundaries for the past, they do not foresee future failure but only help determine when it happens.

There are also Monte Carlo simulations that attempt to vary price in order to see how price distortions affect trading simulations. The idea here is to see how robust the strategy is to market changes and whether it could survive to some shuffling of the underlying trading instrument. These distortions go from  small increases/decreases in the OHLC to bar shuffling and trade skipping. The results show how much the system relies on the underlying market data to provide its result but again, there is no relationship between these and future failure since the market conditions being tested are not known to be market conditions that the strategy will face in the future. There is no relationship between systems that are less affected by distortions and systems that survive under real out-of-sample trading conditions, because there is simply no relationship between these distortions and future market behavior. If future market behavior could be expressed as some sort of perturbation over existing data then we could create trading systems from first principles that would always survive under new data.

A system may show to deteriorate heavily under a Monte Carlo simulation based on distorsions and may yet prove to be very robust. Imagine a system that trades profitably because a company always performs a purchase that moves price after two days of consecutive higher highs, let’s suppose that this has happened and always will happen, the system will always be profitable. This behavior may be constant within the market but random distorsions would make this systems appear to be much worse than a system working on a much less robust inefficiency – say for example an MA cross strategy – because these distorsions only show how precisely the systems rely on past data to look like past data to be profitable, which has nothing to do with how the data will look like in the future. These distorsions favor strategies that apply heavy smoothing and do not rely on precise points within charts, but these strategies really have no higher likelihood to survive.

Selection_999(057)

The above does not imply that Monte Carlo simulations are not useful. On the contrary they can be a wonderful tool to assess statistical strategy boundaries provided that the key assumptions of Monte Carlo simulations are respected and care is taken to avoid cases where Monte Carlo simulations simply fail to provide useful information (I suggest this article to learn more about them). However Monte Carlo simulations are not useful for robustness measurements since behavior under distorsions and worst case boundaries are simply not related to behavior under unknown market conditions — at least as far as I have been able to experiment.

Of course if you can program a Monte Carlo simulation to behave exactly like your trading system, expressing for example money management changes, it could tell you some obvious facts related to robustness – like how all Martingales and grids will with certainty always wipe out an account – but if you’re already coding strategies that have no trade chain dependency and sound trading principles they will do nothing to help you discriminate systems according to real robustness (probability of failure under new market conditions). If you would like to learn more about Monte Carlo simulations and how you too can diagnose your strategies and remove strategies that fail according to the boundaries then define please consider joining Asirikuy.com, a website fiAsirikuy.comlled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.

Print Friendly
You can leave a response, or trackback from your own site.

5 Responses to “Do Monte Carlo simulations say anything about system robustness?”

  1. Geektrader says:

    Hi Daniel,

    interesting article, we do have a “hefty” discussion about it over here, maybe you want to join?:

    http://www.strategyquant.com/forum/topic/4662-monte-carlo-simulations-seem-to-conclude-nothing-about-future-performance-of-a-strategy

    • admin says:

      Hi Geektrader,

      Thanks for letting me know, I’ll keep an eye on it! However I am not a big fan of flaming so I will refrain from getting into the aforementioned discussion. Thanks again for commenting,

      Best Regards,

      Daniel

      • Geektrader says:

        Hi Daniel,

        yes, it´s unfortunate to have these kind of low-IQ-flamers there and that it got out of hand so quick. To bad for the ones that want to have a meaningful discussion about the topic. But yes, I fully understand of course, I´ll be taking myself out of that thread as well most likely, no time to waste…

        Thanks and keep up the good work:)

  2. Rafael Del Rey says:

    Have you read about System Parameter Permutation (or Randomization), by Dave Walton. (http://bettersystemtrader.com/system-parameter-permutation-a-better-alternative/). It works by making minor random changes on setup parameters. According to him, in a overfitted strategy, even minor changes in parameters, cause larger changes in performance. So, by creating a lot of variations, each one with a minor parameter change, it is possible to conclude if the system is overfitted. It is basically, a Monte Carlo simulation where each iteration is produced by changing a strategy parameter (insted of trade order, or skipping trades). I SPP/SPR is right in analysing strategy robustness, I would say Monte Carlo, properly done, has the same properties.

    • admin says:

      Hi Rafael,

      Thanks for writing. The problem with things like parameter randomization and MC simulations is that the area where they provide useful applications is very limited, they are not generic solutions to prove robustness and therefore they cannot be generally applied to any system. They work well when you are exploring a system that in reality represents a highly correlated space of similar systems – when you are trying to judge if something like an indicator period is a good choice – but they do not work well when you are looking at systems for which this assumption is not valid.

      Because of the above parameter randomization and MC simulations will give you valid information when you’re looking at these cases but they will be extremely misleading when you look at other cases. For example if you have a system that relies on an inefficiency that happens at a specific hour and day – say a system that trades an inefficiency based on the NFP – if you consider the trading hour and day as parameters the system will obviously fail in things like parameter randomization because it relies on the very special events that happen in this very specific period of time. Changing the trading time in this case effectively changes the system to something else and MC simulations and parameter randomization will tell you that it’s not robust when it may well be much more robust than other systems in reality (its probability to fail going forward might be much lower).

      My point with this article is simply that MC simulations cannot be assumed to be something generic to evaluate system robustness. They may be useful but you need to really understand when in order to use them. Thanks again for posting!

      Best Regards,

      Daniel

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons