The Kantu Project: Validating a Trading Methodology, System Generation Walk Forward Analysis

During the past few weeks I have been playing with a new program I created for the Asirikuy community which generates systems based on parameterless price action patterns, the Kantu project. On the last few posts I wrote about the basic philosophy behind this program as well as how I implemented pattern filtering, giving you a good idea of where this project is headed. Today I will be talking about what I believe is a very important breakthrough for this program, with the implementation of a system generation walk forward analysis feature that greatly enhances its capabilities. This feature allows users to test their system generation methodologies to ensure that they are able to consistently generate profits in out of sample testing; this implementation effectively eliminates data mining and snooping bias. Through this post I will discuss the problems that led me to the creation of this feature as well as some of the results I have already obtained with it. You will see how Kantu is becoming a very powerful machine for algorithmic trading :o)

Kantu uses a set of candlestick relationship rules in order to generate parameterless systems (no coefficients or indicator period values). These rules can then be combined in a variety of  different ways, generating millions of potential systems that can then be filtered to match some specified performance characteristics. However the ability to generate profitable systems against historical data is nothing new, as this can be done with practically any system generator that has enough degrees of freedom. There is clearly no implicit guarantee of success from having a system with profitable historical results, much more is clearly needed to have a stronger degree of certainty about a strategy’s robustness. The true question is how we can generate systems that can be successful outside of the back-testing data we use to generate the strategies. There are several techniques that can be used to address this problem with the simplest one being in/out of sample analysis.

2-14-2013 12-46-11 PM

When I first implemented the in/out of sample analysis in Kantu I suddenly found myself struggling with the problem of variations in “what worked” through the logic space. For example if I generated 1000 systems using a certain statistical measurement from 1985-2000 and I then out of sample tested the best system from 2000-2012, this top system was sometimes better and sometimes worse in out of sample. This means that there was no clear correlation between my selection methodology and the ability to survive in out of sample, meaning that I was sometimes getting profitable out of sample results simply through random chance. For some symbols this was worse than for others but even on the best symbols I could from time to time find a system with extremely good historical statistics that failed bluntly in the out of sample testing regime. This fact made me very uneasy because it meant that there was a significantly big random component in the out of sample performance of the system generation methodology I was using.

It then came clear to me that – what I needed – is a way to validate the system generation methodology in a way that allows me to be more certain that the probability to have profitable results in the out of sample is high. Since I need to get statistical estimations of how I am likely to do in live trading – which is testing on data not used for system generation – validating the methodology seemed like the best way to go. Validating a system generation methodology is not an easy task as it involves doing a walk forward analysis (WFA) of the system generation process in order to find out if regeneration of the trading tactic every X period is bound to bring profitable live trading results. This simulation effectively mimics what would happen if every given period I would change my trading strategy to match the “top performer” according to my system generation over a certain statistical criteria over a given past period. This is analogous to system WFA, with the big difference that here we are exploring the logic space and therefore we are completely changing our trading technique amongst a very large variety of potential options. In this procedure there are also two rolling windows, a generation window – used to generate a strategy – and a walk forward window where we get trading results of the selected system (the best strategy selected through some criteria on the generation window). These windows are moved until we cover the whole testing space.

As a matter of fact the logic space WFA is a very powerful technique because it answers the question of both “how to trade” and “what to trade” as the run tells you how often it is most favourable to change your system and what system to choose. By validating a system generation methodology you are effectively solving the whole “system creation and rules selection” problem because you now know exactly when to change your system and you know the criteria you need to use to select a new trading strategy. Due to all of these advantages I implemented the above method in Kantu. The implementation is tremendously flexible as you can choose a great variety of optimization criteria for the generation period as well as many other constraints such as the number of systems generated and the setting of a large variety of filters. I also validated Kantu’s simulation accuracy – regular single simulation -by comparing its results with the exported F4 strategies simulated in MT4 (as showed above) to be sure that any results in the system generation WFA (which I cannot carry out in MT4) can be trusted. Note that the differences in the simulation and images are because MT4 contains swap information as well as because the MT4 chart plots trade number while Kantu plots dates.

In the beginning it was hard for me to get profitable results because I was trying to use large system generation periods with much smaller walk forward periods. My attempts to use 5-15 year generation periods with 100-400 day walk forward periods were a complete failure because there was also a significant probability of the selected system to retrace in the walk forward period because – due to the way in which a strategy was chosen – the high profitability and linearity (stability due to the non-compounding nature of Kantu’s tests) seems to give the generated systems a significant probability to retrace. It seems as the market prevents profitable trading when you constantly change your trading system to attempt to match a long term market performer, you are punished because your system’s good historical results make it prone to go into drawdowns soon after you begin trading. I thought that in the case of Kantu using large windows – especially selecting stable systems with large trade numbers – would yield results that would be good, I was mistaken.

2-21-2013 8-46-19 PM

I then thought that selecting systems based on a smaller window might make more sense because systems that are in short term profitable runs might have a high probability to continue these runs while systems that had  been “good” for very long had a propensity to go into a bad trading streak. If a system tests very good during a 6 month window then it is probably going to be good for a little while longer (although not much longer) while a system that has been equally good for 10 years is now “exhausted” and bound to move into some sort of corrective move. By using periods in the 3-13 month range for the generation of a system and then using only from 20-70 days for the walk forward period I was able to achieve reproducible and profitable WFA system generation results. Another important point was the optimization criteria which needs to be a combination of stability related criteria that have to do both with linearity and mathematical expectancy (with ME as exposed by some of Van Tharp’s books). This result is completely amazing – from my point of view – because it validates a whole trading methodology.  With this I now know that I have a good chance of being profitable – in unseen data –  if I change my system every 20-70 days using the past 3-7 months of data using a set of powerful stability criteria. There is a complete absence of data mining or snooping bias.

Kantu seems therefore to a be a big part of our future at Asirikuy because it is going to be fundamental in the validation of trading methods and the generation of trading strategies. Fighting a modern edge war against people with data mining and genetic programming capabilities without these tools puts traders at a huge disadvantage because their adversaries have the ability to conduct unprecedented adaptation and validation of trading techniques while the user devoted to manual system selection is “cursed” with slow development and the inability to properly validate whether the strategy selection method carried out is bound to succeed to changes in market conditions. What does a user do when even a strategy with a successful WFA starts to fail ? The need to completely develop a new system from scratch becomes necessary and the route to do this is tedious and difficult. With Kantu we now have a way to generate strategies on the fly and validate if these generation procedures worked in the past 20-25 years or not.

The first videos about Kantu and its system generation capabilities will come out this week with the first software release coming soon after. If you would like to learn more about system generation and how you too can start generating your own systems algorithmically please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

16 Responses to “The Kantu Project: Validating a Trading Methodology, System Generation Walk Forward Analysis”

  1. Fd says:

    Hi Daniel,

    your method is in fact a subsampling technique: your selection criterion samples randomly from the distribution of all possible outcomes, thus giving you a good approximation to the expected return averaged over the complete parameter space (the pattern rules can also be seen as parameters). My guess is, that your selection criterion for the best performing parameter set (the one you are going to trade in the next period) is of low importance and you will be able to replace this by a less computational intensive criterion, maybe completely random.

    Best regards,
    Fd

    • admin says:

      Hi Fd,

      Thank you for your post :o) Well I am currently filtering criteria based on amount of trades, profitability (just take into account winning systems) and symmetry, then selecting from this group the best based on some stability criteria. Right now I have no idea of whether how I select yields better or worse results (compared to random selection for example) but computationally performance-wise this is not problematic as I calculate all statistics for each simulated system (therefore I have access to them regardless of whether I use them or not). Kantu is very computationally efficient so this whole analysis takes less than 15 minutes on a 25 year period — and I am executing it through the Lazarus debugger which makes it slower. I am sure you’ll have a ton of fun playing with this software once it is released :o)

      Best Regards,

      Daniel

  2. McDuck says:

    Regarding linearity. Walton at Stock Market Wizards speak about the importance of linearity in stocks. This might be another criteria to select robust systems.

    Best

    • admin says:

      Hi McDuck,

      Thank you for your post :o) Well Kantu implements a really cool feature that always shows you the linear regression coefficients of strategies so you can use slopes or errors or sorting criteria if you wish. Thank you for commenting!

      Best regards,

      Daniel

  3. Rick says:

    What is the trade sample for a 3-13 month generation period? Is this daily data you are using? I don’t understand what changes in the forward period. Since these candlesticks have no parameters there is nothing to optimize forward but actually just make another selection. Are you sure you are not projecting gains from a forward period into the next one? In other words, do you close all position at the end of a forward period or you keep the open positions to the next in which case there will be data snooping involved?

    • admin says:

      Hi Rick,

      Thank you for your post :o) This is daily data and the sample varies (as each selection might have a different trading frequency) but I generally filter the systems to only make selections with at least 40 trades. This is also not relevant as what you’re testing is the out of sample outcome. The idea is to in fact test the system selection process, to see if the selection procedure (changing system every X time) actually leads to positive out of sample results. On each forward period you trade a different system selection. This is done in order to create a data selection methodology which we can be confident to use for the future (which we know has at least worked in the past). The selection process is made according to any criteria you want to define. Also there is no gain projection as all trades are closed at the end of each forward period (to avoid data snooping bias). I hope this clears up any doubts :o)

      Best Regards,

      Daniel

      • Rick says:

        So are you saying that you essentially change/select the candle patterns for the next forward period based on data from the previous 20-70 day period?
        If that is the case I wonder is you are ruling out patterns like inside days that occur once in a while but are significant and your apparent success is due to the average gain of random patterns being biased due to the presence of trends. To see if your method works you must either detrend the data or test a market with significant choppiness for a few years. Then you will be able to test your average gain and standard deviation for significance. I am sorry to say but it seems that you have adjusted your forward window to match trend changes in the currency and in this way the average gain from random patterns reflects gains from those trends. I may be wrong but it appear you are doing this. In other words, optimizing from window size is essentially curve-fitting.

        • admin says:

          Hi Rick,

          Thank you for your post :o) This is not the case because the strategies are all symmetric, it means that results are always half long/short trades on every pattern analysis meaning that there cannot possibly be an adaptation to a trend within a currency because we always have equal number of both types of positions with similar profits between both. The strategy is not changing between a net bullish/bearish bias to adapt to the way a symbol is trending as results that do so (which are asymmetric) are not taken into account. I hope this answers your question,

          Best Regards,

          Daniel

  4. Marcelo says:

    WFA is about stability first – not profitability. Stability means a suitable response across different market conditions. It does not matter the widows size but the market conditions in each one, related to your trading scale. I´ll try to explain.
    I try to match different market conditions between IS and OSS windows. I want to know how my setup will behave if I optimize it in a range or trending up or trending down market conditions and put it to trade in a range or trading up or trending down conditions. If I could catch all different matches between IS x OOS windows combinations and the setup shows a stable response across OOS windows – linear growing equity curve, small drawdown, low variation profits, I´ll have enough confidence to trading it in a real account. So, the windows size should have enough space to characterize one specific market (range, trending), considering the scale of your strategy.
    If the setup shows a nice and stable response, no matter what kind of market, you can achieve nice returns with a more aggressive position size.
    A very large window mixes different market conditions. It makes the analysis more difficult because you would have to check if the results are due to a specific condition or a few operations that nailed one favorable condition.
    If you will have bad results in OOS windows it simply means that your setup is not so flexible to adapt to the different OOS market condition. It is not related to the nature of the OOS data, but to the setup. Remind that WFA does not adapt the setup to the next window – because you do not know which market will be. It only helps to test the response of an optimized setup to a specific market condition.
    Sorry about my poor English.

  5. Scalptastic says:

    hi daniel; interesting post.

    i just finished the Michael Harris book; after reading your first Kantu post.

    The book is indeed a gold mine for this sort of work. I see lots of similarities of the logic described above in this post; to the p-indicator mentioned by Harris. is this correct?

    • Rick says:

      Scalpastic, thanks a lot! This is the link I was missing, the p-indicator. There is a post I found in his blog with references to articles with formulas, etc. http://www.priceactionlab.com/Blog/2012/01/what-is-the-p-indicator/

      • Scalptastic says:

        hi Rick; thanks for the links. indeed i see a big potential in the p-indicator type of thing. I reached the conclusion that the p-indicator is no more than a WFA process; and can also be applied to other indicators; not necessarily price patterns only.

        daniel did a good job of pointing to this book; exploring this potential; and writing about it.

  6. Franco says:

    Hi Daniel,

    Excellent idea I love it! Cannot wait to put my new PC to use with this application :)

    If Kantu performance is on pair with Teyacanani and other systems with a larger probability of survival then I see traditional strategies as being obsolete in my eyes. Hate is a strong word but I hate parameters in strategies, they were the biggest headache since I started trading so would be the first one throwing them out.

  7. KP says:

    Hi Daniel,

    Great work indeed! I agree with Franco, will easily replace parameter based system with Kantu generated system assuming their trading characteristics suit my style. Very interested to see the type of DD and DD length of these systems generated.

  8. MM says:

    Hi Daniel,

    Very interesting stuff. Have you perhaps done testing across different pairs, to see whether the results of the above can be replicated?

    cheers,

    MM

    • admin says:

      Hi MM,

      Thanks for posting :o) Yes, but things turned out to be much more complex! I will write about this soon,

      Best Regards,

      Daniel

Leave a Reply

WordPress › Error

There has been a critical error on this website.

Learn more about troubleshooting WordPress.