Unreliability of Backtesting Data… Why you should ONLY use the Alpari Downloaded Mt4 for Backtesting

Through all my time as an automated trading system developer one of the things which I have always dealt with is the problem of backtesting reliability. It quickly became apparent to me that the metatrader backtester had some serious flaws (as evident by some commercial experts which are able to turn dimes into millions in a few years) inherent to the way in which the program does its testing. Besides these flaws which include backtesting interpolation and similar problems I also found out that there were a lot of “data” related problems. Within this blog post I want to talk to you a little bit about these problems and why it is vital to address them in order to have reliable simulations of your trading systems. Particularly I am going to tell you the reasons why backtesting should ONLY be done with the metatrader downloaded from the Alpari website since backtests done on ANY other platform (any non 5 digit broker) are bound to be VERY unreliable, depending on your trading system.
So what are data-related errors ? These errors regarding a trading system’s simulated performance are generated due to quality errors within the historical data used, that is, the data used to do the backtests has errors which do not allow us to do accurate historical simulations. Last week, with the release of the Teyacanani trading system, I received a couple of emails telling me that some Asirikuy members were unable to reproduce my backtesting results. Upon some asking I realized that these members were not using the Alpari downloaded metatrader platform but their OWN broker’s platform to do the backtesting mainly because they believed that this would better reproduce their broker’s “trading conditions”. I was absolutely determined to look into the issue as it was vital for me to know why these results were changing so much.

What I found out was that the metatrader history downloaded on other broker platforms (I tried around 10 different broker platforms with the same results) gives you very inaccurate historical data. At first I thought that the problems were arising due to the appearence of Sunday candles within this historical data (which appear after 3 months of past data regardless of if your broker has or doesn’t have these bars) but then I realized that not only was there an introduction of Sunday candles but there was also a substantial amount of differences with the data of the Alpari downloaded platform. These differences – which I examined from 2005 through 2007 – include the appearence of very low volume daily bars (non-Sunday) which are missing a massive amount of movement with much smaller high/low/open/close values than on the Alpari downloaded platform. Moreover, there is also the appearence of very odd holiday bars which are bars made up of 1 tick with the same high/low/open/close values. The below image shows examples of these errors as well as a comparison between the 5 period ATR on both accurate and inaccurate historical data.

My conclusion from this was basically that this data has a lot of intra-day information missing (which I confirmed) as well as massive amount of inaccurate daily candles, something which inevitably changes the profitability of many trading systems which rely on short indicator periods such as Teyacanani. I was previously aware of these problems with data downloaded through the history center from non-five digit broker downloaded mt4 platforms but the problem seems to be far greater than what I had previously imagined. However some systems are bound to show more or less changes between these two data sets depending on their sensibility.

Through this post I want to warn fellow traders and automated trading system developers that you should ONLY use the Alpari (or other five digit broker) downloaded metatrader 4 platform for backtesting and that using any other broker for backtesting does NOT make the backtest approach your “broker’s conditions” , on the contrary, the downloaded data may introduce contradictions as Sunday candles on a non-Sunday candle broker and all the errors I mentioned before. For this reason- if you want to backtest your systems with reliable data- download the metatrader platform from Alpari and do your backtests there. I have compared this data with Oanda Tick data and it seems to be in fair agreement (while the other data is NOT) so again, ONLY use a five digit broker metatrader platform to perform backtests.

Previously I had written that the metaquotes platform should be downloaded but they recently changed from a five digit (in build 225) to a four digit (in build 226) broker by default making the platform download the bad data. Alpari US/UK (which was the previous broker used) does download reliable data.

This Sunday I will be releasing a video on Asirikuy about all these issues and the extent to which they affect automated trading, including changes with Teyacanani and other experts detailing all the possible effects these errors may have. If you would like to learn more about automated trading and backtest reliability please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

4 Responses to “Unreliability of Backtesting Data… Why you should ONLY use the Alpari Downloaded Mt4 for Backtesting”

  1. only says:

    Hello, thanks for your post.

    I would like to kindly ask you something.

    I downloaded 9 years m1 alpari data from

    http://alansforexblog.com/2010/02/10/how-to-set-up-metatrader-history-data-and-get-90-backtesting-quality/

    backtesting results where pretty impressive.

    Now when i compare them to the result from alpari metatrader data downloaded from the history center , i got totally diferent results(both are five digits data), to the point that my “profitable strategy is a loosing strategy now. The tests of course where done for the same period.

    Why you think this may be happening? bot are al pari data..

    i have been forward testing my strategy on alpary metatrader for the past two weeks, but cant tell yet wheter its profitable or not.

    would be happy to hear from you,

    regards.

  2. admin says:

    Hello,

    Thank you very much again for your comment :o) I have never used the data from alansforexblog but I can tell you that the data mentioned on this article is also unreliable as within 2010 a lot of gaps where formed on the data. Note that the data download on the Alpari UK history center is NOT Alpari data but data from the Metaquotes servers. This data has seen the introduction of many errors from when this article was written.

    Since this data could not be trusted I finally was able to obtain reliable 11 year data from a source directly within Alpari (friend of mine) in csv format. This data is available within the Asirikuy community in MT4 hst format. You can read more about this here : http://mechanicalforex.com/2010/10/solving-backtesting-problems-finally-a-true-absolutely-reliable-data-set-for-asirikuy.html .

    Since I got this reliable data I have never looked back to the History center or other online data sources. Definitely using accurate data is a MUST to obtain reliable simulations. I hope this has answered your questions. Thank you very much again for your email,

    Best Regards,

    Daniel

  3. only says:

    Thanks for you fast answer. Still something very strange is happening, lets not talk about thhe quality of the data,

    i downloaded 3 different metatrader 4 (from metatrader website, alpari and fxsolutions.

    I used the same data from the website mentioned on my previous post, and i got toally diferent results in the three cases, different trade numbers, profits etc. Do you think this is only due to the spread ? i m really confused.

    thanks for your atention

    • admin says:

      Hello,

      Thank you for your comment :o) As I tell you I no longer use data from any metatrader server or website but the csv data I obtained DIRECTLY from a contact in Alpari which I know is reliable (NOT from their severs or from any website). I also use a spread changer program I coded to always backtest at the same spread level so I never have any such problems with spread dependency. It is very easy to get very confusing results because many data sources (MT4 paltforms and websites) are unreliable and if you add the different spreads/swaps assigned on different broker’s demo accounts you end up with tests which are close to useless because they are always different.

      As I have told you within Asirikuy we use a reliable 11 year, 1 minute dataset with a spreadchanger program which guarantees that we have as little variation as possible on our tests. I hope this helps you out :o)

      Best Regards,

      Daniel

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons