Walk forward analysis (WFA) is one of the four important development fronts we have been tackling in Asirikuy. This technique – which validates optimization procedures through a repetitive optimization/forward testing cycle – is bound to be very useful in the creation of more robust trading methodologies (if you would like to learn more about WFA, please search this blog for previous posts on the subject). However, the WFA methodology does make some assumptions that you need to take into account if you want to have the highest possible chances of success as ignoring them may cause some important problems. On today’s post I am going to explore how the use of WFA behaves relative to different data sets and how this translates to the practical use of WFA in live trading.
When doing the walk forward analysis of a trading strategy we use a long term data set that we then use for all the optimization/walk forward cycles. For example, we take 12 year data from a reliable source and we then perform 2 month optimizations followed by one week walk forward tests, this would give as a total of about 620 walk forward periods which – when joined together – give us the overall result of the walk forward analysis. In the case of a profitable WFA, the strategy has shown that it has been able to survive a 12 year period of optimization/forward testing that validates the optimization technique used. If – on top of this – the strategy has enough degrees of freedom as to satisfy adversarial trading (completely contradictory logic is possible) then the system has also demonstrated adaptability.
–
–
However we need to take a deeper look into what the strategy has demonstrated. Does this mean that the strategy will always be able to survive in the long term if we optimize for two months and then walk forward for one? The back-testing procedure implies that this was true for the back-testing data during the past 12 years but it doesn’t talk about the ability of the system to survive walk-forward analysis on a completely different feed. It may well happen that a system may be able to have a profitable WFA using feed A but when using the feed from broker B this may not happen. It is also true that the walk forward analysis assumes that the optimization and walking feeds are one and the same and NOT different. This last point is very important as it implies that we should always use the same data feeds for optimization and walk forward.
But how bad can this problem be? Well, I did a small experiment in which I performed a 2 month rank analysis optimization of a system across 6 different brokers and I found that the trading results for the six were fairly different. Although they were fairly well aligned across some variables – such as the general strategy used and the overall TP/SL values – it was clear that the best trading scenario had been slightly different depending on the broker. In some cases the difference was also quite dramatic and the number of trades in the optimizations changed significantly. I repeated this test for several different time frames and I also found that the broker became worse as the time frame dropped. While results were almost identical on the daily time frame, they were in some cases dramatically different on the 30M and 15M time frames. This is obviously because broker dependency increases very significantly as you reach into the lower time frames.
The most interesting thing also comes after you do a walk forward using the above optimization sets. The results always seem to be worse when you use the optimized set from a different broker, perhaps because the optimization to a particular data feed does fit to some peculiarities of the particular data used. Although these results are highly anecdotal – far more evidence is needed to draw anything conclusive – it does point to the possible conclusion that using the exact same data feed in walk forward analysis might be quite important (although probably not critical). It is very conceivable that the performance of a trading system might degrade very significantly if the optimization portion of the analysis is done on a completely different feed.
It is also possible to evaluate this on the whole WFA analysis, although the dynamics are much more complicated. For example you can perform optimizations for all the separate optimization periods with one data set and then load all the indicated parameter sets to walk forward on another data, this will give you an idea of how much the results of a system can change in the long term if you optimize on one data set and then run your system on another. I have not carried out this analysis yet – due to the obviously higher complexity – but this would definitely give some hints into possible problems when doing something like this.
–
–
In my view, the best thing to do is to preserve the assumptions made in the WFA – which uses the same data for optimization and walk forward – and perform the same procedure when live trading. This means that if I am implementing the procedure used in WFA on live trading, I would optimize the system using my own broker’s data feed instead of some other back-testing feed. This will make sure that I adapt my system to quirks of my particular data which would also reflect the “adaptation to quirks” present in the WFA. The F4 strategy tester and WFA modules will use live broker data for the practical last step optimization used to derive the evolving parameters needed for live trading. Optimizing with another data set and then walking forward on another broker’s data is a scenario that is best avoided.
Obviously if you have any information that shows the above to be untrue for one or several strategies, please feel free to share on the comments. Clearly the importance of the above problems will depend on the time frame and trading strategy used but it is safe to say that using the same data for optimization and walk forward at least preserves the assumption made in the WFA analysis. If you would like to learn more about WFA and why this technique might lead to more robust strategies please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)
Hi Daniel,
your findings are a clear indication that our current procedures for optimization – as well as the criteria for engagement – have a bias towards noise.
The central problem I’ve been struggling with since my first attempts to put together a profitable EA is the period length used in almost all indicator: how much of historical data should you use to infer your trading decisions?
If we take for granted that quotes consist of a signal and a noise component we should focus on eliminating noise in indicator measurements. Feeds between different brokers should only differ in the noise component, thus having indicators robust to noise should help to eliminate broker dependency as well.
Best regards,
Fd
Like Fd mentioned, if a system is robust and is using sound trading principles, it shouldn’t be bother by the noise generated by using different feeds.
Don’t you think you’ve over-optimized the system to only “curve fit” certain market with the limited use of only that particular feed? I like to see many long term trading systems that use the same (un-optimized) parameters for 20-30 years and are still performing well with relatively low risk. Examples are like Long Term Trend Trader John Henry and Bill Dunn.
Sorry, I trade more futures than in Forex markets. Correct me if I am wrong.
Another good article, thumbs up Daniel.
Hi Angel,
Thank you for your post :o) Well I was talking mainly about the WFA procedure which assumes that the same feed is used for optimization and walk forward analysis, since optimization in WFA is generally carried out for small periods of time, it makes sense that the process is sensitive to quirks in the data. There is no inherent curve-fitting in WFA because this involves a validation of an optimization/walk forward procedure, hind-sight in this case is quite limited.
Certainly it is also possible to get systems that can achieve 20 year results using a single set of parameters but there is obviously a larger curve fitting component than in a system that reached a 20 year profitable result through WFA. When optimizing systems – no matter how long the optimization period is – you are never validating the optimization/walk forward process so the ability of your optimized parameters to adequately forecast good trading results remains untested. A 20 year optimization process could simply be considered as a very long WFA optimization period, but proper walk forward validation steps would still be needed and sadly we’re limited by the volume of data available. You should also consider that there is no such thing as “unoptimized” parameters, if you have a parameter selection you have selection bias, not choosing the best possible selection is simply under-fitting (getting a worse result on purpose). Making a smarter selection – based on rank analysis for example – is a much better solution than simply not bothering with selection (which is very sub-optimal). The best thing obviously would be to lack the need for selection (systems with no parameters).
In my view WFA has the potential to lead to much more reliable historical testing statistics – from a robustness point of view – although some care must also be taken into account to avoid potential pitfalls. Thanks again for your comment,
Best Regards,
Daniel