Developing a trading methodology: Why choosing a single top system is a bad idea

During the past few weeks I have been immersed in the study of several basic trading methodology questions that I can now research thanks to the development of the Kantu parameter-less price action based system generator. From all of these questions perhaps the one that interests me the most is how you can develop a system selection methodology that has a high probability to generate profitable results in out-of-sample testing. To answer this question we must learn how often we need to generate new systems and how we can select the systems most likely to perform profitably in out of sample from a given poll of strategies. This is no easy task as we need to understand the dynamics of system generation as well as how in sample variables relate to out of sample performance in a way that is not dependent on a single symbol or time period. Today I will share some of my research, particularly how traditional system selection methodologies give poor results and how our view needs to expand beyond single selection in order to achieve very good out of sample performance.

Let’s start by explaining the problem a little bit better. Suppose you have to start trading today and you need to come up with a strategy that will be very likely to generate profitable results during next year. The most intuitive way to solve this problem – and the one with the most assumptions – is to generate a trading strategy that has had very good historical performance and then choose this system assuming that the system will maintain the same behaviour, at least to some extent. But how do you know that this will work? The fact is that – since we cannot know the future – trading any system is like taking a gamble because your strategy can enter a deep drawdown or it can even fail completely if market conditions change enough. How do you analyse this problem quantitatively and come up with the real best solution?


Kantu allows me to explore this problem through an analytical walk forward system generation procedure. I can suppose that I was facing the same problem as described above (but in the year 2000) and I then carry out the process of generating a new system, selecting a system and trading for the next year during each year till 2012. I use the past 10 years of data to generate my strategies, testing different filters for the generation as well as different criteria for the final selection of the strategy I will be trading for the next year. After each year my results are added up and in the end I can get an estimate of how I would have really done if I had followed my procedure. This has no bias and emulates the procedure I would have gone through if I had really started trading in 2000. With this tool we can now run several tests and see what the best way to carry out this procedure actually was.

The first thing that I can say is that using the traditional intuitive approach has a high probability of not working very well. You might be profitable in the end – although your probability to do so is not very high – but you will always be below the statistics of the top performer you chose. Chances are you will not be so lucky and you will be netting a loss through most out of sample period, leaving you with a net loss after ten years of trading. The reason why this happens is because you cannot simply predict the out of sample results of a strategy and, regardless of the criteria you choose for filtering or system selection, there is always a chance for that top performer with astonishing historical results to turn sour. Choosing a non-top performer (for example picking randomly from the top 5) doesn’t help you too much as well as all of them have a chance to fail their statistics.

This was very intriguing for me as it pointed to the fact that beating the out of sample was not possible because of the inherent randomness in the future. However I then decided to take a deeper look and analyse the populations as a whole, for example what are my chances of generating a system with a profitable out of sample if I select only highly profitable or highly linear strategies. The results were encouraging and demonstrated to me that – in the cases I studied – filtering the selection did improve my chances of having a profitable out of sample. Instead of having the 45-55% chance of hitting a profitable out of sample result when using no filters (such profitability above zero) I got chances in the 60-80% region when using certain filters. However this wasn’t enough because you still face the problem that whichever selection you make can become a disaster. Even with these filters you still have a pretty good chance of making a bad string of selections and being in a drawdown after ten years of trading (and I don’t want to take that chance!).

Perhaps there is a way to select strategies that will eliminate this chance? After testing many different types of sorting I was unable to find any in-sample statistical parameter that guaranteed this, there was always a chance – due to the fact that the future is unknown – that we would get into a string of bad system selections that would take years to recover from. Then I realised that the problem wasn’t the filtering or selection criteria, it was the fact that I was using a single selection. Since some filtering criteria do increase the probability of a profitable out of sample and we know that a single selection can always turn sour, the solution here is to avoid the single selection by using at least 5-10 top performers. Incredibly when you take a single system there is always a chance for disaster but when you trade the top 10 performers – under certain selection criteria – you always seem to be able to come out profitably in out of sample. Even though one or two might be unprofitable, the edge given to you by your filtering and selection criteria makes the probability of all of them failing very low. The selection criteria is in fact very important here because it gives an additional edge to the top X performers (they tend to have a higher average out of sample profit than the average of all generated systems). 

2-26-2013 1-15-36 PM

After all of this I think I have now come to an important answer to a basic question I have had during several years. Is there a way to generate and select strategies that gives a very high probability to have a “good” out of sample result? The answer seems to be that there is and it relies on choosing basic filters to attain high quality systems in the generation step, coupled with some criteria for final system selection that give you an edge in the out of sample when you trade the X top performers (5-10 in my tests up until now). Obviously I am only touching the tip of the iceberg now and my conclusions might change with the additional research I am carrying out but certainly the results up until this moment are encouraging and point to a light at the end of the out-of-sample tunnel.

If you would like to learn more about my work in automated trading and how you to can learn to develop your own algorithmic strategies please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

12 Responses to “Developing a trading methodology: Why choosing a single top system is a bad idea”

  1. Franco says:

    Interesting finds, I have always believed that the more strategies you run in parallel the better your overall chances of having a profitable portfolio is. That is the reason why I like to run portfolios with 10+ systems but very low risk for each system to have a total risk of what you would use for one system. Of course this requires a lot of capital but with a micro account you can get away with $6000 or so.

    The question now is can the selected systems be compiled as one system? That would make admin work a lot easier!

    • admin says:

      Hi Franco,

      Thank you for your post :o) Sure, since there is no risk of over-optimizing these systems (because they lack parameters) you can in fact take advantage of the F4 framework’s ability to have several systems running on the same EA. We could in fact build a single EA containing all the needed instances to trade a Kantu generated portfolios. We could even have F4 parse the system rules from a database filled by Kantu (why not!), once we figure out the practical details of how we can trade with a high chance for OS profitability — the sky is the limit,

      Best Regards,


  2. Igor says:

    Hey Daniel,

    What do you think about the idea of taking the top X systems and turning them into a comittee?

    For example you get 9 systems, and if 5/9 are long you go long otherwise if 5/9 are short you go short.

    The idea is that if several of the systems are unprofitable they will get out-voted by the others that are profitable.

    This would work well if you dont use TP/SL or closing patterns so that the systems are always in the market.

    • admin says:

      Hi Igor,

      Thank you for your post :o) Very interesting idea! Although it is a little bit difficult to evaluate. I’ll leave in inside my idea cabinet for the near future ;o) Thanks again for contributing,

      Best Regards,


  3. Dz says:

    Exceptionally great article – something everybody is thinking of, but nobody can measure.

    And opens answers to lots of intuitive strategic questions for future studies: compounding, game theory, etc. etc. which are all becoming measurable now….

    • admin says:

      Hi Dz,

      Thanks a lot for your comment — I don’t get the words “exceptionally great” too frequently ;o). You’re right in that the power we have now is Kantu’s ability to evaluate things in a more quantitative manner, certainly we can now evaluate methodologies in real trading data, an extremely powerful tool (in my opinion). Thanks again for posting,

      Best Regards,


  4. Jan says:

    Hi Daniel,

    I enjoy reading your articles. They lead us to the core of strategy development.

    Have you considered researching the correlation between the complexity of a strategy and the OS performance? Complexity can be measured as the number of comparisons (eg. C[0] > L[1]), or the number of bars involved in the strategy. There could be a optimum complexity a strategy can have, because too little complexity gives a “under-fit”, and too much complexity a “over-fit” strategy.

    Greetings, Jan

    • admin says:

      Hi Jan,

      Thank you for your comment :o) I’m glad you enjoy the articles! As you say my interest is precisely to go into the very core of strategy development. Your complexity theory is also quite interesting, we have long thought that less complexity means better OS performance but now we can actually quantify whether this is or isn’t the case for Kantu’s generated systems. Thanks a lot for the suggestion,

      Best Regards,


  5. Scalptastic says:

    hi daniel; exceptionally great post again..:)

    I dont want to be negative; but wanted to share my experience; so other traders can benefit from it; or at least have it the other way.

    In my research of trading strategies; i have spent numerous time and efforts on various system generation ideas; and tried few commercial software too; and did encounter problems such as the one mentioned in this post; how to select a system from the hundreds of generated systems with *good looking backtest curves*.

    Sadly I have come to the conclusion that this path did not work for me personally. Here is why:
    – Trader need to bond with the trading system being traded. meaning you need to know the in/out of it in all details.

    – Any good trading system; needs constant monitoring and improving; day in and day out. Every trade needs to be analyzed and improved; be it the entry/exit methods, execution, broker…etc.

    – You only need a one profitable system; or perhaps a couple uncorrelated ones; and concentrate on them.

    – It takes almost a couple of years to improve a system to a solid profitable system; after constant trading, monitoring, improving, trading..etc.

    I had my share of automated generated stuff; and gave it up all. I havent looked back since; and quite happy with the couple of systems I have. One of them been in development for over 2 years.

    It is funny how it all happened; I did have the ideas for both systems back then; but never able to get them constantly profitable in the market. After few months of trial and error; got it finally to be profitable and very happy with the performance.

    – auto generated systems are too simple; they mostly have the entry/exit and stops defined; in a simple way. There are all sort of nitty gritty details that you have to do to make a simple idea works.

    Take for example the Turtle system; does it work? of course it is STILL profitable. However implementing the text book rules will not be profitable. You have to do the extra work of customizing it to the changing market condition.

    – specialize or generalize: Any pro trader will tell you that; developing profitable systems takes years. That is because you need to specialize one idea; and spend time to get it right. On the other side; if you get 1000 systems that are generated; analyzing one by one properly; in terms of bonding and understanding why it works or does not work; will take months.. that is not feasible.

    You are better off picking an idea and specialize on it; and spend months; until you get it right.

    – Sleeping well: you will not sleep well if you have auto generated systems trading live; because you don’t know what’s expected.

    hope this help you guys; and sorry Daniel if I have been negative on Kantu or any auto generated tools.

    btw; I do use the auto generation from time to time; but just to get new ideas; but never use the systems straight out.


    • admin says:

      Hi Matt,

      Thank you for your post :o) There is no problem in pointing out what has worked for you and what you believe is needed for successful trading. I certainly agree with you in that anything you use to trade successfully – being manually or automatically generated – needs to be fully understood. Perhaps the issue you have had with automatically generated systems is that you lack enough understanding of the generation process and therefore this lack of knowledge about “what is expected” in terms of out of sample performance, has meant that this approach simply does not work for you, while the manual generation/adaptation process is something you really understand and feel good with. Certainly this approach won’t work for everyone – it is not my intent to make it so – but what I want to do is to build a complete understanding about the automatic system generation process and how it can be improved to make it work better (through judicious sampling and statistical analysis). Clearly this may be used in different ways by different people and using software like this to get “ideas” to then polish into better strategies is also perfectly valid. So long story short, I am not trying to impose this technique on people as being better for them – it might or might not be depending on the level of understanding that is achieved – but I just want to share the results of this research and where it might lead :o) Thank you very much for sharing you thoughts and ideas,

      Best Regards,


  6. Bruno says:

    Hi Daniel! Thanks for this great article :)

    I thought the general recommendation was to run no more than 4-5 Asirikuy systems. Now is that recommendation changed? Would it be advisable (or not frown upon) to run one instance of each system, each with a very low risk setting?


    • admin says:

      Hi Bruno,

      Thank you for your comment :o) This is still very early research so we should avoid drawing any conclusions or taking any action before we confirm the results. This is also only pertinent to the price action systems generated by Kantu and we cannot make the leap to assume that it would also apply to any general combination of systems. In the meantime all the recommendations we have are still valid but obviously they may change if the evidence points to ways in which we can trade more systems without greatly increasing risk. Thanks again for posting :o)

      Best Regards,


Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons