To Stop or Not to Stop: When is it the Best Time to Stop an Automated Trading System

Like it or not, all algorithmic systems will eventually fail.  Since a system generates drawdown under certain trading conditions, it is warranted that a string of market conditions can exist such that the system will take the account to zero. It is not a matter of “if” it is simply a matter of “when”. Obviously when designing trading systems we make a big effort to make the system withstand both market evolution and past market conditions but it is also true that we cannot fully prepare for what is essentially chaotic and largely unknown. Considering this, it becomes key to know when a system needs to be removed from an account because it is no longer showing “good” results. This issue is very delicate and difficult to address because sound statistical tools need to be used in order to determine the best rational way to deal with this situation. Through the rest of this post I will talk about this issue as well as what I now consider the best way in which to manage this situation.

Lest us consider that we have an account in which an automated trading strategy is being executed. This system has a projected maximum drawdown of 10% during the past ten years and a maximum number of consecutive loses equal to 10. Then – a few months after we start trading it – we reach a drawdown scenario of 15% and we start wondering if the system still works as it “had worked” during the past. How do we know if this system should be allowed to trade or if the algorithmic strategy no longer works ? Provided all  past simulations are accurate, should we remove the system because it went above what historically was considered the “worst” drawdown?

The answer is probably no, and relates to the way in which strategies work. Trading systems – even those that have some discernible statistical edge – will experience ever increasing drawdown excursions in the future, merely because the probability that we will experience market conditions that are different and unfavourable increases. Montecarlo simulations perfectly show the above point, if you run a 5, 10 and 25 year Montecarlo simulation, you will see that the 25 year simulations will always have deeper drawdowns, even if the core distribution of returns is still exactly the same. Even if your system uses some form of dynamic adaptation – such as walk forward analysis or neural networking – the algorithm will eventually fail (because an nth degree adaptive mechanism is still dependent on adapting to the past, the way in which you adapt can also become obsolete). There is no absolute protection – from the system design perspective – against system failure, because the nature of the market (semi-chaotic) causes it (unless the market could be demonstrated to be totally deterministic, something all statistical market analysts I know agree is not the case).

After discussing the above, it becomes clear that the first thing we need to know in order to stop a system is to  know when a system has failed. This is no easy task because there is no absolute definition of what “failure” is. Failure can mean wiping an account for some (so you never stop trading) while for others it relates to other arbitrary ideas (such as a 10% drawdown with no statistical basis). If you take the statistical point, then failure could be interpreted as when the system shows statistical characteristics that deviate – within a tight confidence interval (say 95%) – from what has been deduced from Montecarlo simulations of the strategy. If a system betrays its own statistical nature, then it should be removed from trading.

The problem with the general statistical definition – either when defined through Montecarlo or through more formal hypothesis boundary statistical means – is that the point of failure of systems is too deep because systems will take long to show they are not to be trusted. Imagine if you owned a supermarket and you were grading your fruit suppliers according to how many “bad fruit” they got you, if you decide to cut suppliers only after they drop 50% below their 10 month average “bad fruit” count you will have to endure a ton of “bad fruit” before you can tell them they are not needed anymore. If their count starts dropping slowly, your tolerance will also drop as the average will get lower and lower. In Montecarlo simulations a similar thing happens, the statistics of systems evolve towards worse values if they deteriorate slowly and you can only exit fast if the systems fail quickly.

The best way to stop or judge system failure is not the Montecarlo approach, in fact, the Montecarlo approach can be considered one of the most primitive formal solutions to trading system management. But if we do not want to wait for systems to fail statistically then how do we stop them ? The answer to this question – which is now what I believe is the best approach to use – comes from the area of game theory. The most powerful concept is that you never stop systems, you just reward and punish them. This solves the issue of when to stop systems, the issue of when systems fail and the more important – and often neglected issue (even in my own blog) – of what to do after a system is stopped.

Granted, the above doesn’t come without a price. The application of game theory is focused on the reduction of regret on the worst possible case but obviously if you were facing a best case (all your systems were going to work perfectly for the next 20 years) then game theory will diminish those profits because it will hinder trading from systems that were not going towards failure. In essence you are sacrificing return on your possibly good cases in exchange for a clear boundary for when things are bad. Using the game theory approach, total portfolio failure will end up with a much lower drawdown – when compared with the Montecarlo scenarios – and with all systems not taking any trades (although the worst cases for the game theory approach will be a subject of a future post, think game theory Montecarlo simulations).

The application of the game theory algorithms and the design of the “ghost” history keeping databases is also not trivial and currently something we are working on (and in fact close to testing, at Asirikuy). If you would like to learn more about our community and how to get a deeper education in automated trading please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

2 Responses to “To Stop or Not to Stop: When is it the Best Time to Stop an Automated Trading System”

1. Chris says:

Hi Daniel-

Welcome back and its great to have you back full time.

This is a very interesting topic that I’ve struggled with myself. Agree that waiting until beyond the worst case scenario has occurred is a sure way to guarantee you will never make any money or at least never take any profits.

I agree that varying the lot size, or not trading a system at all based on some other criteria is the way to go. For a system with a winning record, it will surely lower the profits, but its an acceptable risk if it protects our capital from loss.

Another way to look at it is, what other investments or asset classes are competing for the investor’s cash? Every asset class has a recent and historical return and returns on Equities, Bonds, Commodities have been doing well recently as the central banks create more currency out of thin air. Its race to inflate our assets because eventually we are all going to get crushed by inflation due to all this money printing.

If we had a way to calculate the risk-adjusted rate of return for each asset class or trading system, then the decision would be much easier. And based on that, we can decide if its worth putting money to work in a trading system or asset class.

Again welcome back and let us know your thoughts,

Chris

Hi Chris,

Thank you for your post :o) Sure it’s good to be back!! I am still not “fully back” (still moving in) but my level of activity has indeed increased exponentially within the past few days (and I am glad it has!).

About the MC worst case, I do not agree with you in that you cannot make profit if you wait for the WC, we have in fact seen several cases where people have been able to exit portfolios at the WC scenario with a good profit. I would say that one does not necessarily exclude the other. However it is true that if a system stops working, you will have to pay a higher cost to find out if you wait for the statistically defined scenario. This has to be done because if you quit earlier you are simply making a wrong assumption (you are assuming it stopped working when statistics tell you there is a high probability that it will work going forward). The game theory approach solves this problem, it is also important to note that this does NOT involve lot size variations because this leads to a granularity problem. Using lot sizing is NOT a proper solution to the fade in/ fade out strategy.

About competition for cash, I agree. We do in fact have the Martin and Sharpe ratios to give us an idea about the risk adjusted statistics. If we adjust these statistics to account for the statistical worst cases you will have an idea of whether the risk/reward scenario makes sense against another asset class. The problem of course is that you will make comparisons historically and this may not mean anything when going towards the future. In 1998 a risk-adjusted analysis would have lead you to invest in the S&P 500 right before the dot com bubble, so I would say that if you’re going to invest it is better to invest in things where you understand the risk (where it comes from and how bad it might get in the future) rather than compare between different asset classes where the risk is not clear. My personal policy is to invest in whatever I understand best.

I hope the above gives some interesting insight into my opinion on these matters. Thanks again for your comment Chris :o)

Best Regards,

Daniel