It is no mystery that one of the most important challenges in the development of trading systems is to be able to get a good estimate of future risk and profitability. The first approach to do this is to back-test the strategy across a large segment of time (usually 10 to 20 years) but this approach is plagued with problems that become more and more apparent as the degrees of freedom within trading strategies start to increase. Within this post I want to talk to you about some of our experience in the area of trading system development, why the above approach is problematic and why walk forward analysis can help us eliminate many of the above mentioned issues.

Our journey starts with a very important question. How do I know the amount of profit and – more importantly – risk that I should expect from a trading strategy in the future? The obvious answer would be to carry out simulations of the strategy over a long period of time in the past and then determine – through these results – what the expected boundaries for profit and risk might be going forward. The problem with this approach comes when you consider the inherent plasticity of strategies, natural to the degrees of freedom contained within them. How do you know if your strategy matches some ever-present aspect of the market or if it just happened to profit from randomness in the market through what is nothing but random chance?

–

–

It is important to notice here that I have not even mentioned optimization. This is because the above problem is inherent to all systems, even if you decide not to optimize them at all. The fact that there is some inherent parameter selection in your strategy – even if you just don’t get it through optimization – creates a selection bias which gives you a probability of having a strategy just fit to the random motions within the market, achieving profit out of some aspects of the market which might not be robust enough for trading into the future. The most important problem in the creation of trading strategies is to be able to know if your strategy is in fact able to survive to future market conditions, if your strategy is able to overcome the power of the selection bias. What you need is to know the expected characteristics in future trading, outside of any in-sample trading.

The first way to do this is simply by adding an out-of-sample test to the end of your testing period. If you optimize a strategy for 8 years and then perform a back-test for the out-of-sample period you could get some information about the robustness of your strategy by watching how the strategy performs during the period for which there was no information available for selection. This in fact gives you an idea about future performance. The problem – and a big one it is – is that the spectrum of acceptable conditions within an out of sample test is huge unless the out of sample test is tremendously big. For example if a strategy is in-sample optimized for 8 years and then out-of-sample tested for 3 years, is the fact that the last out of sample year was unprofitable enough reason to avoid trading? The fact is that this could just be normal behaviour of the strategy because it might be within its acceptable statistical limits.

Due to the above it becomes clear that out-of-sample testing in this way is not enough to establish survival to selection bias. How do you do it then? The answer – made famous perhaps by Robert Pardo in his book ” Design, Testing and Optimization of Trading Systems” – is to do a walk forward analysis in which we derive our statistics from the construction of many in and out of sample tests. In the walk forward analysis routine we use a “moving window” of optimization and then we use a “forward window” to evaluate our strategy, at the end of each forward window analysis, the optimization moving window is slid and a new forward period is evaluated. In the end the merging of all the forward periods builds a long term test which is – for all practical purposes – entirely out-of-sample, there is in fact no parameter selection bias and these results can be used as a valid expectation of future performance (to a very good extent).

However things are not as easy when it comes to out-of-sample testing. The first thing that you will find is a good deal of disappointment relative to in-sample conditions. The main indicator by which a system is measured – when doing walk forward analysis – is the walk forward efficiency, which compares the profits of all forward periods against the profit of all the optimized periods. A ratio of 1 or higher implies that the system is exactly the same in in-sample testing as in out-of-sample testing (meaning that your selection process is perfect) while a lower ratio indicates that you lose some profitability. However the worst case is when your ratio is negative, meaning that you are just chasing some aspects of the market which are very variable and from which you will never be able to profit from (because – as a dog chasing its tail – you’re always too late to the party, trying to cash on some phenomena which happened yesterday but will not repeat itself going forward).

–

–

There is another caveat to walk forward analysis which is the selection of optimum parameters in the optimizations. Which is the best method to select parameters? Most profitable, least drawdown, highest profit to drawdown ratio, most winning trades? Well, experience has showed that the most significant way of doing this is actually *not* by selecting any particular statistical value but by imaging the whole parameter space as a function of a profit to drawdown statistic and determining which areas of the space are the most “flat” (meaning have least variation). By building an n-dimensional topological map of system parameters and evaluating the statistical characteristic as a function of “parameter proximity” we can generate a selection of results which has a very good probability of performing better.

It is important to realize that the above selection process is fundamental because otherwise the achievement of profitable walk forward strategies becomes extremely hard as the number of degrees of freedom grows too large. For example a strategy like Teyacanani – with very few parameters and a limited number of degrees of freedom – can give walk forward efficiencies of even 50-80% using simple “highest profit” as an optimization selection criterion while a system like Atipaq – which has about three times more parameters – struggles to get above 0% using this same selection process. As the degrees of freedom get larger, so does the need to evaluate the entire parameter space of systems in order to carry out the “best parameter” selection in the in-sample part of the walk forward process.

Sadly when developing strategies using walk forward analysis the options are quite limited because programs such as MT4 include no walk forward capabilities and other software packages don’t give you the needed freedom to explore the adequate system selection processes (as in MT5). Therefore the best option is to develop the back-testing, optimization and parameter space evaluation inside your systems in such a way that your systems will be able to adequately go through this process regardless of the trading software you are using.

In Asirikuy we aren’t using walk forward analysis up until now, but it has become clear that we want to move in this way and our implementations of internal back-testers, optimizers and parameter space analysers is on its way. The F4 framework has opened up the doors to the use of tremendously fast internal ANSI C back-testing capabilities, something which will give us the ability to optimize our strategies within the back-testing process (running it normally in MT4 or MT5) being able to also select parameters in whichever way we want. Although the challenges remain to build adequate parameter selection algorithms – potentially the topic of a future post – we currently know that all the tested Asirikuy systems can give profitable walk forward efficiencies and it will therefore be a very big step forward to move in this direction.

If you are currently developing trading systems I would advice you to perform walk forward analysis to see what the potential of your strategy might be (and to obtain much better risk and profit targets). It is also very important to reduce the number of degrees of freedom if you want to get away with using simple selection criteria or to come up with more complex methods to select best parameters if your degrees of freedom are high. Another important point is also the choice of sizes for the moving in-sample window and the forward window, something which we will be discussing on a later post. If you are using MT4 you might also consider using this software (not affiliated in any way with me or Asirikuy) to perform very simple walk forward analysis although your systems will need to be compatible with open price simulations to give any meaningful result (meaning that they should give reliable tests using only OHLC data of previously closed bars).

As you can see walk-forward analysis is a very powerful stress-test for your systems and it definitely opens up the way for the elimination of the parameter selection problem in trading system creation. If you would like to learn more about the things we have developed in Asirikuy and gain a deeper understanding about algorithmic trading please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Hey Daniel,

Excellent post! Walk forward analysis sounds like the ideal way to move forward but implementing it as you said is going to take a lot of effort and brain power, not to mention computational power.

I would love to see how fast F4 can backtest strategies when done internally, if it is fast enough to analyze a big portion of the parameter space in a realistic amount of time then we definitely have the “grail” as traders out there would call it.