Judging trading systems using Monte Carlo simulations: A second look

For the past 7 years I have been a strong advocate of using Monte Carlo simulations to evaluate on-going trading system performance after their creation. These simulations allow you to expand on the possible scenarios that can be derived from a system’s distribution of returns, allowing you to basically see a wider variety of potential scenarios that a system could generate while remaining true to its distribution of returns under in-sample conditions (read more here). However after two years of dealing with relatively large portfolios and after gathering substantial amounts of evidence my opinion on this subject is starting to change, given the assumptions inherent to the use of Monte Carlo simulations and the reality of algorithmic trading system performance under real out-of-sample conditions (meaning performance under market data that didn’t exist when the system was created).

The main assumption you make when using Monte Carlo simulations to evaluate trading performance is simple: trading performance under unknown market conditions will follow the same distribution of returns as the distribution generated by in-sample back-testing. In essence you are expecting your distribution of returns in the future to be the same as your distribution of returns under in-sample conditions. If the system is tackling a real historical inefficiency in the market – meaning it’s not the product of statistical bias sources – and the market continues to contain this inefficiency in the future, then it should hold true that your system should generate an equity curve that spawns from the back-testing distribution of returns. If it doesn’t, then it should be discarded.

Of course, life isn’t that simple. There are several problems with the assumptions above that make it fairly clear that the distribution of returns in the future will not be a copy of the in-sample distribution of returns and that expecting that is naive. Since historical data is not infinite and market evolution is not bounded, then variations of the market in the future that are unaccounted for in the in-sample data will present themselves. Furthermore, the fact that historical inefficiencies that were not apparent in the past are now apparent means that their exploitation will be attempted and therefore the market structure will change accordingly. Even if the strategies generated contained absolutely no curve-fitting or data-mining bias and all inefficiencies found were guaranteed to be historically real and statistically relevant inefficiencies, their mere existence implies that market participants will act on them, which changes market dynamics to something that hadn’t happened in the past.

This means that any system found using historical data will be subject to some degree of edge distorsion – changes in performance under new market conditions – that will depend fundamentally on the magnitude of statistical bias sources present within the system’s creation and how market changes going forward affect the underlying inefficiency the system is trying to exploit. This does not mean that all systems will perform worse – some indeed might perform better – but the average expression of this edge distorsion will be negative and therefore it can be thought of as a decay. In stock trading this is usually called “alpha decay” but in forex trading, we’ll simply call it an “edge decay”.

Given that the distribution of returns under new market conditions will deviate to an important extent from the distribution expected from back-testing – in a negative way on average – it is worth asking whether Monte Carlo simulations are an ideal tool to diagnose system success or failure. They will basically tell us to stop trading every system we find sooner or later, as this decay is expressed, limiting the potential use of systems. Monte Carlo simulations also make it impossible to restart trading of strategies that are discarded as under this paradigm a strategy that is found to behave outside of its in-sample distribution of returns simply needs to be discarded. If such a strategy is to be traded again then the in-sample distribution of returns is accepted to deteriorate and the Monte Carlo simulation serves no purpose as this completely opposes the fundamental assumption behind its use (to ensure a system is behaving within in-sample expectations). If you are going to just restart trading something that has been proved to statistically mismatch its in-sample distribution of returns then just don’t use these simulations as you basically are negating the relevance of their conclusion.

I believe we now have better tools to assess whether to trade or stop trading a strategy and to resume trading a strategy if this is presumed to be favorable. Since all we care about is the expectation of future profitability – not necessarily whether in time a strategy always behaves in line with its in-sample distribution of returns – what we can do is use more advanced mechanisms to obtain probabilities of success going forward and only trade strategies for which these probabilities are favorable. Using the machine learning algorithms we have developed to draw out-of-sample profitability expectations (see here) we can naturally stop trading systems for which no positive performance is expected and we trade strategies for which profitable results are expected. There is no permanent discarding of systems but all we care about is whether an algorithm can be statistically expected to perform profitably going forward.

This fundamentally changes the conversation from a past-looking conversation – have we performed as well in the recent past as we expect from in-sample data – to a forward-looking conversation, whether we expect to make profit going forward from an algorithm, regardless of the past. I believe that this approach is much more direct and better satisfies the need to focus on the most important goal, which is to have algorithms that have positive expectations of profitable performance going forward. If you would like to learn more about how you can also use thousands of algorithms to trade using this kind of approach please consider joining Asirikuy.com, a website – not a holy grail – filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.strategies

 

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons