Seasonality and Walk Forward Analysis: Not All Systems Are Created Equal

On a recent post I discussed the occurrence of seasonality effects within trading strategies and how these effects can be studied in order to improve the trading characteristics of trading systems. Briefly, seasonality effects are changes in a system’s statistical characteristics that happen periodically (for example that a system has a higher probability to lose on trades taken on Monday than Friday). These changes need to be identified to be statistically meaningful in order for system modifications to warrant a smaller degree of curve fitting. We discussed how a robust number of trades are needed and how deviations and distributions could be used to tackle whether or not a seasonality effect exists. Granted that this approach only works for seasonality effects that are broad (such as all Mondays, all Augusts, etc) it doesn’t work for seasonality effects that are too specific (all December Fridays on a US election year) and still, 25 years of data are needed to gain higher certainty about broad seasonal effects. Today I want to talk about a very interesting effect I found that relates walk forward analysis (WFA) with seasonality, this seems to be the answer to why some systems show unfavorable WFA on the same window as others show favorable WFA without any simple explanation.

One of the biggest mysteries I have come across during the past few months has been the difference in WFA results for different trading strategies. For those of you not familiarized with WFA, this is a technique that attempts to validate an optimization procedure by running a sequence of optimization and walk forward (out of sample) steps. The optimization window is shifted by a given step size and the walk forward windows are “stitched” to achieve a long term test (usually 10-20 years) that shows you how the optimization procedure of optimizing across an X sized window and then running the system for a period Y works. In summary, this technique allows you to validate the first degree adaptive capacity of your system (provided the system has enough freedom in its variables) across along term period. Please search this blog for “walk forward analysis” for more information about this technique.

1-9-2013 7-51-22 AM

Going back to differences between systems in WFA, some strategies seem to be able to give profitable WFA with very small windows (just a few months or even weeks) while others need year sized windows in order to yield any profitable result. This seemed a little bit puzzling as it is very interesting to note how two strategies trading on the same time frame – with similar 25 year profitabilities achieved in rank analysis optimization – can somehow behave so differently when you try to make them adapt to evolving market conditions. At first glance I wanted to blame degrees of freedom for this sort of problem but after finding strategies with similar degrees of freedom that showed the same behaviour I was completely puzzled.

The easy answer – what many would give at first glance – is that the strategy that achieves survival on the small window scale is simply a better system because it tackles some rather more fundamental market behaviour (a behaviour so fundamental that you can always adapt to changes within it and it remains fairly constant). However if strategy A has this behaviour but strategy B can work profitably with a larger window across more symbols, then strategy B might tackle a more fundamental behaviour. However why does strategy B need a much wider selection of market conditions than strategy A if the exploited inefficiencies happen within the same time frames, using similar amounts of data? The answer seems to come from seasonality camp.

I have found that strategies with seasonality effects need larger optimization windows than strategies without seasonality and that strategies that have fast repeating seasonality (such as a strong propensity to have losses during a certain day of the week through the whole year) have worse overall results unless this seasonality effect is accounted for. However when discussing WFA we must also consider that we are discussing seasonality across the entire parameter space, something much more difficult to tackle than seasonality across a single parameter set. If the parameter space of a strategy has a tendency to show seasonality then you will find that randomly chosen parameter sets across the parameter space will have a tendency to show the same negative changes across the same periods. If the parameter space has randomly distributed seasonality – you aren’t able to identify any periodical statistical change along the space – then you will not have profitable WFA (this is a conjecture I’m making).

Why do strategies with parameter space seasonality require longer WFA optimization windows (at least the cases I’ve studied) ? The answer could be related with the system’s attempt to improve conditions during the seasonal effect across the WFA. If a strategy has a tendency to have losses during May and you run an optimization during May and trade June, you will find that June will always be a losing month. Since May will always be a losing month – because April’s optimization is always going to mismatch due to the seasonal effect – you will find that the number of losing months will increase very significantly due to the large effect of the seasonal effect in the WFA. Including longer optimization periods effectively “dilutes” the seasonal effect such that you are optimizing to the yearly “average” that includes the best possible case during the period where you have this effect.

1-9-2013 7-53-15 AM

However if your seasonal effect is on a faster scale (all Mondays tend to be unprofitable for example) you need to effectively avoid trading on Mondays in order to greatly increase your WFA. Since losses on Monday are a characteristic of the parameter space, we can assume that filtering Mondays is a valid characteristic for this system because the inefficiency it trades has some reason to become non-existent during this day. It is no coincidence that several momentum following techniques have bad Mondays, it’s mainly because signals obtained during Monday also depend on Friday’s momentum which has already happened two days ago, in essence Monday trades are created with information that has a two day lag, therefore the measurement becomes less valid. Fast seasonality requires elimination while longer term seasonality can be accounted for by increased WFA optimization window sizes.

Now another interesting case is the lack of reproducible seasonality along the parameter space, meaning that your strategies tend to give very different seasonality depending on the parameter set you are choosing but there is no clear path in which the strategy is heading when you evaluate this effect. When this happens I have seen that WFA results are poor and they often tend to not be profitable at all. This means that the strategy’s ability to adapt in short term windows is very small because the parameter space for the strategy is probably too incoherent. This means that the parameter space seems to eventually cover some other trading inefficiency different from the one you are trying to exploit. This clearly happens when you have a logic space (rather than a simple parameter space) because you’re trading a “space of spaces” that encompasses many different trading strategies. However it can also happen on complex strategies where changes in parameters can imply complete changes in the trading method used (for example large degrees of freedom in time filtering). You can achieve profitable WFA in these systems by restricting the parameter space so that it tackles the adaptation of a coherent inefficiency (reduce the parameter space to one of coherent or no seasonality).

Overall, seasonality effects are very interesting because they tell us a lot about a trading strategy and the actual nature of what we are trading. These effects become even more interesting when you look at how they are distributed across entire parameter spaces because you are able to distinguish drastic changes in the inefficiency being traded. If seasonality can be considered a “mark” for a given trading method, then we could use this technique for other purposes as well. The best thing seems to be to have a strategy without any seasonality but if such an effect exists we should take advantage of it in order to improve the trading results of our systems. Are all the above described conjectures actual facts about seasonality and WFA ? Much more analysis and examples are needed before we can draw such conclusions. Please feel free to share your opinion with me within the comments :o)

If you would like to learn more about trading systems and how you too can learn to improve your system design skills please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

 

Print Friendly
You can leave a response, or trackback from your own site.

6 Responses to “Seasonality and Walk Forward Analysis: Not All Systems Are Created Equal”

  1. Fd says:

    Hi Daniel,

    I think using WFA to draw conclusions about the validity of filtering out trades depending on some criteria is a very indirect approach. I’m not saying it can’t work but I would prefer to calculate it directly. A way to tackle this in an automated manner would be to compare several General Linear Regression models. Each condition can be modeled by a dummy variable and you will choose the model with minimum AIC.

    Just to illustrate the perils of a trading based evaluation approach:
    The analysis for 1 EA I tested showed that Wednesdays seem to be bad days for trading. Eliminiating them however showed 2 additional loosing month which have not been there previously. Doing it the other way round first eliminating the loosing months showed Mondays with losses. But eliminating additionally Mondays was finally a satisfying solution because all numbers improved.

    Best regards,
    Fd

  2. fxbootstraps says:

    This is an interesting article. One idea that I had was to work with parameterless queries on the ea logic in SQL Server and then just run through wheels within wheels.

    This is one of the reasons why I joined asirikuy as I found that while this looping idea worked well within SQL Server I need to understand the analysis better.

    Also I found this free resource that might be of use to members wanting to know the basics about statistics too.

    https://www.coursera.org/

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons