## The Frequency Distribution of Returns : Developing Further Criteria to Determine When a System Stops Working

If you have read my previous posts on Monte Carlo simulations and the analysis of monthly returns you may have already realized that a very interesting part of automated trading is to know when a system “stops working”. Definitely one of the reasons why inexperienced traders in this field tend to “change systems” and dump rules when they enter draw down periods is because they fail to have an understanding about the natural deviations from long term statistical outcomes they should expect and this in turn leads them to be uneasy about any moves into unprofitable territory because they lack any criteria to tell them when such moves communicate that a system has stopped working.

There are obviously many ways in which we can determine that a system has stopped behaving as we expect it to behave. During last week I wrote a post about Monte Carlo simulations and how these allow us to know when a system’s short term statistical outcomes are deviating from the normal deviations we would expect from the long term statistics we obtained from our back tests.  However Monte Carlo simulations carry several limitations, amongst them the fact that we need to reach certain draw down depths (usually twice the ten year historical draw down for systems with risk to reward ratios near 1:1) or long periods of testing (several hundred trades) before we can know if a system has stopped working.

To compliment Monte Carlo based simulations, it becomes important to develop further criteria based on the distribution of trading returns in order to know when our “return pattern” is becoming very different to the pattern observed during the past 10 year period. An Asirikuy member recently suggested we incorporate this as part of our worst case analysis, and I believe that this has great value in providing us with key insights into the way in which our systems behave and when they stop behaving as they should.

The idea here is quite simple. If a system behaves like it has behaved during the past ten years, then the distribution of returns going forward will look like a random “pick” from the previous ten year distribution. That is, if the system had for example 100 trades within 0.5-1% profit and 200 trades within 1-2% we would expect the next 50 trades to be a “random pick” from within this previous ten year distribution, holding true to the above mentioned values within the possible randomness of the pick. The above image shows the distribution of returns of the God’s Gift ATR within its 2000-2011 backtests. From this it seems clear that any random pick would be constituted by a large majority of -0.5% losses with some small amount of small profits and some smaller amount of  >5% profitable trades. This analysis of return also shows us that this EA with this settings is bound to be very hard to trade psychologically since trades have a high probability of being individual losers (although the system is overall profitable because many profitable trades are 5x to 10x the average losing trade).

Statistically speaking, what we want to do is determine if the probability that a given trade set is a random pick from the previous ten year distribution is enough to consider this trades a part of the previous set or not. If the trade set we evaluate seems to be likely to belong to the previous 10 year distribution then the system is behaving as it should while if it is not we have a dramatic change in behavior which tells us that the system is not working as we expect it to anymore. Long story short, if the trades taken within a live account are distributed in a way in line with what would be expected from the past 10 year behavior the system can continue running while if it is not, the system must be stopped.

Now, in order to know if the above is true or false we need to use some statistical tools that will help us compare and see if our system’s live trades do follow a return distribution as we would expect. To do this we use a Pearson’s Chi-square test which allows us to compare the results of our “new set” with the “reference distribution” and determine whether or not our outcome can be thought of as a “random pick” from the previous ten year distribution.

This can certainly sound complicated to those of you who are not very familiarized with statistics but the fact is that all these calculations will be incorporated into an Asirikuy new “worst case EA” which will allow you to evaluate any system and its current stand against its “worst case” comparison. The EA will compare the current EA statistics and “distribution of returns” with a “moving window” in such a way that it will alert you whenever a system falls below its predicted thresholds determined by Monte Carlo simulations or if its distribution of returns appears to have deviated significantly from what was expected from the ten year analysis.

The frequency distribution of returns and its comparison with live trade sets using a Chi-square test is certainly another very interesting and useful way to evaluate a system’s worst-case scenario in such a way that greatly compliments and enhances the information given by the Monte Carlo simulation thresholds. If you would like to learn more about system evaluation and how you too can design and evaluate your own likely long term profitable systems please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach to automated trading in general . I hope you enjoyed this article ! :o)

### 4 Responses to “The Frequency Distribution of Returns : Developing Further Criteria to Determine When a System Stops Working”

1. Maurizio says:

Dear Daniel, thank you for your recent efforts to define the risk of our systems and portfolios, which is an essential aspect of trading. Trading a portfolio allows to better manage the overall risk by eliminating a system which reach the worst case scenario and adding another a priori profitable one. Do you think it would be possible to code, into the same EA code or separately within a specific indicator, an alarm which will advise us when a system within a portfolio or the portfolio itself will reach the worst case scenario, in order to stop trading it? It could be for example an indicator that could give for each system in a portfolio and for the whole portfolio different data as total profit, maximum DD, monthly profit, monthly DD, maximum reached DD in relation to the worst case scenario and if possible also data related to Monte Carlo simulation and frequency distribution of return compared to the historical distribution. What do you think?
Kind regards.
Maurizio

Hello Maurizio,

Thank you very much for your comment and kind words :o) Have you been reading my mind ? I am currently developing an indicator that will allow us to automatically control Monte Carlo based criteria within our systems so that we can know when a system should stop trading. The indicator will display trading result distributions, maximum draw down, consecutive losses and other criteria for separate systems within a portfolio. The indicator will also have a “database” in which it will hold the worst case scenarios of all Asirikuy systems derived from Monte Carlo simulations, warning the user when a Monte Carlo worst case has been found. Currently I am also working on distribution tests (such as the Pearson chi squared test mentioned on this article) and possibly during the following few weeks I will implement this additional criterion into the indicator. Besides providing us with VITAL information about our systems this indicator is also going to be some pretty eye candy (at least I am trying to make it so !). Thank you very much again for your comment,

Best Regards,

Daniel

2. Al says:

Hi Daniel,

I think that’s a great direction, since it can allow us to stop trading a system before it gets to the WC scenario, thus hopefully minimize some of the loses when a system has stopped working.

I have a question though: how would you regard a system which is starting to behave outside its normal expected behaviour, but to the profits side? :-)
Or should I put it differently, should we also have a “Best Case Scenario”?

While it is tempting to think things just ‘got better’ for us, it does require heavy checks because if it is not working as expected, to either side, we need to at least update our calculations, and perhaps modify risks assumptions or something else.

Perhaps a little bit hypothetical :-0, but still I think we should address this.

Thanks!

Hi Al,

Thank you very much for your comment :o) I am glad that you like this approach in determining when to stop trading a system. Definitely there will be sometime before we can implement such distribution tests but certainly within the next few months we might have some interesting preliminary results. The main problem here being that some systems are simply not fit for such analysis, systems like Ayotl which have few but very profitable trades do not give accurate tests because the chi-squared evaluation breaks down when trade frequency is too low or zero on a given class.

Regarding the profit side, I believe that the distribution analysis may also give us an idea of when a system is becoming “less profitable” and therefore we can in this way reconsider if the risk we are taking is worth the profit we are receiving. However this “stop trading because of too little profit” scenario is quite dangerous since systems can and do have shown in tests that they can be barely profitable to breakeven for a prolonged amount of time (sometimes even 2 years) only to get an extremely large winning streak after that. In my mind evaluating draw downs and preserving capital is the most important aspect, evaluating how a system behaves towards the profit side should be done with great care as the evaluation can yield extremely skewed results depending on the amount of time in which you do it. However the Monte Carlo criteria already includes a criteria for profitability, in the form that if after X trades the system is not within profit then it fails to comply with its long term statistics as all systems with an edge should reach profit after a certain number of trades if they comply with the values derived from long term simulations.

Regarding the “best case scenario”, this would be very dangerous and just a “fantasy”. In my mind it is NEVER good to prepare people for the best as the psychological pressure when dealing with draw downs will become greater and the “betting” mentality of running systems just because “the best case scenario is so good” would certainly take place. I – as many other professional traders – like to think only about the worst things that can happen and prepare for them, hoping to get any scenario better than that. Of course this is pessimistic but creates a “shield” around short term emotions and protects me from having expectations that may cloud my judgement or get me “overly excited” about a certain possible outcome which may just be a “small probability” (like what calculating a “best case scenario” might be).

So, long story short, I believe that we shouldn’t calculate any profit expectations besides those given by simulations but we should in fact stress systems to the maximum in order to find worst case scenarios as “worst” as we can. Evaluating distributions and how they compare with their historical counterparts might be a good way to do this and determine whether a system is “failing” from either a profit or a draw down perspective. I hope this answers your questions :o) Thank you very much again for your comment,

Best Regards,

Daniel