Trading System Walk Forward Analysis: Are you sure you’re adapting?

During this year one of the main points of focus in Asirikuy has been the development and analysis of systems using walk forward analysis techniques. This algorithmic system development method – popularized by Roberto Pardo’s books – attempts to develop trading strategies using an evolving procedure in which the parameters of a trading strategy are determined according to a given moving window in the past (if you want to learn more about walk forward analysis, please search this blog for previous post on the matter or refer to Pardo’s work). The main idea of walk forward analysis is to demonstrate that the system can survive parameter changes brought forward by changes in market conditions. However, little attention is usually paid to the meaning of positive walk forward analysis results, especially what they really tell you about the system you are trading.

Most people are just happy to get any sort of profitable walk forward results and this – without a doubt – is a positive sign for any trading system. However the big question is whether the use of the walk forward analysis procedure is justified for a strategy because you can be using a more complex analysis procedure without any true benefit when compared with a good fixed optimization method (such as methods that optimize parameter set stability in the parameter space). The key here is that the walk forward analysis procedure does not bring any benefit in virtue of itself, unless the adaptive qualities of the procedure are truly tested out.

In order to understand how a walk forward analysis can yield results that are no better than a fixed optimization, we must first learn how to analyse its results. The first thing we need to consider is whether a trading strategy has a very good probability to be profitable regardless of the parameter selection used. For some systems – especially those based on few variables that tackle longer term movements – most parameter sets yield results that are somewhat profitable. In this sense, the probability to get a profitable result out of random chance is significant and therefore we could indeed expect to have a long term profitable result just from random selection, regardless of any walk forward analysis procedure.

As you can see, one of the key aspects in determining the relevance of the walk forward analysis is in comparing the result with the potential results obtained from a random selection of parameter sets from the parameter space. In this sense the main idea would be to perform many tests using randomly selected parameters for each walk forward period and then compare their statistical characteristics (profit, drawdown, etc) with those of the results obtained from the walk forward analysis procedure. In essence what you are doing here is a Montecarlo simulation of the parameter selection methodology in order to determine what the probability of generating your walk forward results from random chance actually was. For some strategies you will see that the probability to achieve something close to the WFA results is high while for others it is clear that without the WFA procedure the results couldn’t have been achieved with a significant probability. In this cases the use of a WFA procedure brings no discernible advantage against a usual optimization because you cannot see a clear difference between adaptation and random chance.

Let us suppose that you have obtained results showing that the probability to obtain profitable WFA out of random chance is very low, this still does not mean that you are getting better results than a normal optimization because this still doesn’t show that your system is adapting in a relevant way. Suppose that you have a system that has profitable WFA results, the question now becomes whether those WFA results are merely a consequence of rather constant parameter selections or a true consequence of a larger degree of adaptation. A problem with the WFA procedure is that it doesn’t provide any clues about adaptability if the need for adaptation through the test is low. For example if a system faced rather similar conditions through the whole testing period, then its ability to adapt hasn’t been tested.

The way to evaluate this is to look at the way in which parameters have varied through the whole test in order to see the degree of adaptation that the system faced. It is also important to see if the system only had profitable results under certain regions of the parameter space, something that clearly shows that the system didn’t do any good job at adapting when conditions changed but merely made profit when market conditions were fit to a certain region of the system’s parameter space. This analysis might reveal that the regular optimization result is even better because it may show that it is better to trade a single parameter set because the system simply lacks any ability to meaningfully adapt when market conditions change beyond the scope of the inefficiency it attempts to exploit. When analysing walk forward results it is important to make graphs plotting parameter variations against forward period profit to drawdown ratios (or some other relevant statistics) in order to see if there is clustering amongst certain regions. Remember that you want a system like a cockroach – that has survived successfully under many environments – and not like a giraffe, that has survived successfully for a long time but has never faced the need to adapt to very drastic changes. 

In essence what I am attempting to say here is that walk forward analysis is no holy grail. You cannot simply trust the results of a walk forward analysis as a fail-safe that a system can successfully adapt to changing market conditions because dynamic selection and adaptability are not the same thing. Although the walk forward analysis technique allows a system the possibility to adapt to changing market conditions, it is up to the trader to determine if the system used this capability through the analysis and whether or not any true adaptation was in fact present. There is definitely a lot to explore in the WFA arena besides what is generally described in algorithmic trading development books :o) (expect some examples of the analysis mentioned through this post in future blog articles)

Right now we are close to the first testing of a new and powerful strategy tester module in Asirikuy which will enable us to test a lot of theories and carry out a lot of analysis around WFA which is impossible (or at least tremendously difficult) under MT4. If you would like to learn more about walk forward analysis and how we are developing these ideas please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

 

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons