Are Our Assumptions True ? : Validating the Ten Year Back-testing Hypothesis

As a scientist I was trained to evaluate hypothesis and stress-test them against any possible flaw in order to see whether or not my assumptions were valid or invalid. Contrary to the way in which many people seem to develop systems – simply disqualifying other’s views and negating and discrediting any testing that might prove their assumptions wrong – I like to work from the basis that what I want to achieve is long term profitability and any evidence that shows that what I am doing is wrong should be considered seriously as I’d rather be wrong than unprofitable in the long term. From this point I have always been a little bit uneasy about a certain assumption we have made within system development that seemed to have no evidence basis from a “practically proved” point of view. This assumption is that 10 year RELIABLE and ACCURATE profitable backtests lead to a higher probability of succeeding under future market conditions in the long term. On today’s post I will share with you my first true evaluation of this hypothesis, how I was able to test it out and what my finding were.

I thought for a long time that we would never be able to evaluate the 10 year back-testing hypothesis from a practical stand point until we achieved 10 years of live testing which we could compare, something which would allow us to say : ” yes, 10 year profitable backtests increase the like hood of achieving profitability in the future 10 years”.  The main reason why I thought this evaluation would not be possible is that intra-day data for currency pairs is only available – at the earliest – from 1998 which is the year when the widely used electronic brokers started to come to live and the first true “modern forex” began to appear. Before this time only EOD (end of day) data was logged (simply the open/high/low/close of each candle) as trading was mostly confined to banks and a large number of non-retail speculators who didn’t trade through brokers but directly through inter-bank servers.

The answer to my evaluation dream became clear when I began experimenting with reliable control point simulations and systems developed within their limitations. Suddenly it became absolutely clear that we could perform reliable testing – if ALL limitations are adequately taken into account – using only EOD data in Metatrader 4. Knowing about this I decided to look for some EOD data for forex pairs to see if I could get a hold on data going back to at least the early ninety’s so that I could perform a ten year system development followed by a 10 year “out-of-sample” evaluation.

Luckily I was able to find a few sources of EOD forex data which allowed me to have data for some pairs going back to at least 1978, although the earliest data only contained high/low/close information which did not allow for accurate simulation. However I did find a few pairs with data going back to 1990 in which open/high/low/close EOD data was fully available, something which allowed me to get a true evaluation of our 10 year backtesting hypothesis.

What I did next was quite simple. Take 1990-2000 data on a pair and develop a series of profitable systems that give accurate control point simulations, then evaluate these systems from 1990-2010 and see if during the 2000-2010 period their profitability changes dramatically (if they fail) or if they continue to give profitable results. The above represents exactly what we are doing in Asirikuy, take 10 years of data to develop systems and then attempt to run those systems for the next 10 years.

Results were surprisingly good and in line with what we expected. Systems that give profitable simulations from 1990-2000 on the tests I did (mainly on the GBP/USD and USD/JPY), always gave profitable results on the 2000-2010 period, sometimes to a larger and sometimes to a lesser extent but they did not turn into account wiping or dangerous systems. Systems that were not profitable in the 1990-2000 period sometimes gave profitable 2000-2010 period but never to a larger extent that what would have been expected from chance.

These finding are very important because they constitute the first true evidence we have that – in the forex market – long term evaluations do increase the probability of having success within at least the next following 10 year period. Above you can see a typical graph of a system developed during 1990-2000 and evaluated then from 1990 to 2010. The results show that the strategy manages to survive to a 10 year out-of-sample testing, revealing that the assumption that strategies developed within a 10 year period have a higher chance of working later on is true, at least for these systems and therefore at least not proved false in general. This also hints to us that the phenomena has some validity, that there is some hard data to believe that a 10 year framework development based on back-testing is bound to work in the long term and adapt to changes in market conditions.

This weekend I will do an Asirikuy video explaining this evaluation procedure more in-depth sharing other results on other currency pairs with fellow Asirikuy members (plus information about the data source, etc). Another interesting fact that arises from the above is that we can now develop systems for some pairs over a 20 years – instead of a 10 year – framework (however only reliable control point simulation strategies). Certainly this will give rise to some interesting development and discussions within Asirikuy. If you would like to learn more about my work and how you too can design systems with long term profitability and robustness in mind please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach to automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

WordPress › Error

The site is experiencing technical difficulties.