Solving Backtesting Problems : Finally a True, Absolutely Reliable Data Set for Asirikuy

One of the main problems we have experienced when using the Metatrader 4 platform is that the quality of the data downloaded from the history center is doubtful. Metaquotes recognizes that this data is “indicative” – which means it is gathered from several different sources – implying that its not as trust-worthy as data which comes from a particular broker feed. The nature of this data makes several problems arise to a various extent on different currency pairs with the EUR/USD being the most “reliable” data set while others like the USD/JPY and NZD/USD have serious data quality problems. On today’s post I will share with you some of the problems of the metaquotes history center downloaded data (the 5 digit broker data set which is the most “reliable” one) and how I finally managed to obtain some truly reliable broker data which will be used from this weekend in Asirikuy.

But I thought you said the data was good ? True, I had performed an analysis of the EUR/USD data from January 2000 to January 2010 using excel and I had found almost no holes in data with only some very small problems attributable to the missing of some small parts of the data (just a few hours through all the set). However during the past few weeks I have been encouraged by a few Asirikuy members – who have brought some problems to my attention – to analyze other currency pairs and the quality of the data from January 2010 till June 2010. The results were quite surprising showing that Metaquotes data has some big and serious issues on this year’s data and particularly on previous years on other currency pairs.

What are these problems ? Well, the main problem we have is that data is missing. The overall quality of the data that is available is good but there are several “batches” where there is no data. There are several BIG problems with this issue as missing data damages the quality of the tests – not only for that period – but for every period that uses this data to derive indicator data. For example, a few days are missing in March on the EUR/USD data, this means that all 14 period ATR calculated data will be different for at least 15 days, simply because the previous data (which is needed to calculate the real ATR value) simply does not exist.

The EUR/USD however is a particularly bad example to point bad data quality because the pair has good quality for almost the entire past 10 years on the Metaquotes 5 digit broker data. However, other currency pairs don’t have the same luck and problems appear as very large batches on previous data. For example, the USD/JPY pair is missing almost the ENTIRE 2004 year (see image above). It therefore becomes a big problem because there could be an important over or underestimation of long term profitability and draw down characteristics depending on the nature of the results for that year. The fact that the systems in Asirikuy use money  management that depend on the outcomes of previous positions also makes this quite important since results in the future might be different simply due to the loss or accumulation of profit in previous years. The NZD/USD is another good example of such a case with about three months of data missing in 2008.

After performing a systematic analysis of consistency for the 6-8 pairs we frequently use in backtesting in Asirikuy it became obvious that the quality of this data is not good enough to have reliable backtesting results. Even though Asirikuy systems are robust and the profit and draw down targets might not change a great deal for even a missing year it becomes absolutely clear that the long term analysis of profit and draw down might have been deviated from reality. For this reason I decided to solve this problem once and for all.

The first idea to solve this problem was to buy data for all the currency pairs. Reliable 1 minute data with no holes for a 10 year period costs about 1.5K USD per currency pair meaning that we would need about 5-6K (with some discounts for block purchasing) to get all the data we needed. This option was consulted within an Asirikuy forum thread in which members started to vote whether or not we wanted to make this purchase as a community. However, I started to look for other options as the idea of a community purchase didn’t seem that achievable within Asirikuy (at least in the short term).

Finally I decided to email all the brokers I have “big” accounts with to try to get some files with their historical data. As a client you may have a “right” to data access on your broker so I realized that I had absolutely nothing to lose by asking for 10 year data from all these brokers. My request was answered positively on a few brokers but most offered me only data for the past 2 or 3 years, something which is not good enough to run tests of long term profitability. In the end Alpari UK was the only broker that allowed me to get almost 11 years of data (January 2000 till October 2010). This Alpari data is direct bid data from the broker with NO holes, NO errors and no problems like the metaquotes indicative data. I have run tests on all the currency pairs and I am glad to say that this bid data has amazing data quality, coming straight from a well known broker source.


After receiving this data I promptly converted it to Metatrader 4 format and used it to rerun all Asirikuy backtests. All the systems remain profitable and all the profit and draw down targets are similar although some systems do show better while others show worse results. The above image shows you a comparison of the USD/JPY backtests for Atipaq with metaquotes and Alpari data. You can see that 2004 (the year previously missing) was actually a pretty good year. This weekend I will be launching a few videos regarding this topic (updating backtesting tutorials) as well as the whole set of new backtests. The data will be available for download (in already converted mt4 format for all the different currency pairs) within Asirikuy and the tutorials will explain how to load it to run reliable backtests.

I am happy to say  that this will be the last time I will write a post about backtesting reliability. The new data I obtained for the Asirikuy community shows no holes and will allow us to develop systems without any doubt about their results under previous market conditions (provided that they are built in a way that doesn’t exploit the limitations of the back-tester!).

If you would like to learn more about simulations and how you too can develop your own systems that are back/live testing consistency within broker dependency please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

14 Responses to “Solving Backtesting Problems : Finally a True, Absolutely Reliable Data Set for Asirikuy”

  1. Maxim says:

    Daniel,

    Yesterday you’ve talked about whether trading is art or science.
    This theme can be debated a lot. I think that it is certain that the approach to trading must be scientific.
    To my mind, your article is a perfect proof to my statement.

    Maxim

  2. Maurizio says:

    Hello Daniel, thank you very much for your last post and for your contribution for reliable simulations and backtests. I saw that, despite Atipaq is still as profitable than before on usdjpy, the equity curve shows clear differences. Maybe this would be even more evident on other systems or pairs. Do you think it would be better to re-optimise all the Asirikuy systems on the real data, in order to have the best profit/DD ratio?
    Thank you.
    Regards.
    Maurizio

    • admin says:

      Hello Maurizio,

      Thank you very much for your comment :o) Interestingly enough I attempted to reoptimize several of the systems (Teyacanani and Atipaq) but my results kept on being the same. It seems that the systematic errors previously introduced did not cause a lot of difference in this regard. This falls in the line with the fact that settings tend to converge towards a given set of parameters as the number of years increases, for most systems removing even a year does not take them too far away from their current optimal values. However as always I am open to debate and system improvement so if you are able to reoptimize them to better values let me know and we’ll open up the discussion within the Asirikuy community forum. I encourage you to follow what you think can be done to make things better and share your results :o) Thank you very much again for your comment :o)

      Best Regards,

      Daniel

  3. Vincent says:

    Will the historical data updated from time to time? I had downloaded from Asirikuy.com

    • admin says:

      Hello Vincent,

      Thank you for your comment :o) Sure, I will keep the data updated in 6 month intervals. I will however update it again on January so that we can have full 2010 backtests. I hope this answers your question !

      Best Regards,

      Daniel

  4. Füge says:

    Hi Daniel, I like Your approach to Forex. Please write the exact steps, how can I get the history data (1 Minute or Tick data). How much money is it? What is the format of the data? HST? How can I use it in my FxPro Metatrader 4 program to test my EA correctly? Thanks

    • admin says:

      Hello Füge,

      Thank you for your comment :o) I am glad you like my approach to trading ! The historical data is Alpari data available within Asirikuy (you would need to become a member to get it), it is 1 minute OHLC data (not tick data), it is in HST format (directly loadable into MT4) but I only recommend using it with Alpari UK since otherwise candle timestamps, etc will not match and you might get skewed results. Thanks again for your comment,

      Best Regards,

      Daniel

    • Titia says:

      With the bases loaded you stcurk us out with that answer!

  5. Nico says:

    Hello Daniel, I just discover your site and I must say it is one of the best on automatic trading.

    I’m interested in Alpari datas and of course be a member of Asirikuy but I have few questions:
    – how many currencies do you have?
    – what is the time frame of the datas, are they really minute datas (it’s wonderful if it is)
    – and the more important for me: do you update these datas?

    I’m tired of looking for good datas and your solution sounds good!

    Thank you!!!

    Nicolas.

    • admin says:

      Hello Nico,

      Thank you very much for your comment :o) I will now answer your questions :

      – There are 7 different currency pairs, each one with at least 10 years of data
      – The data has a one minute resolution
      – Yes, I update it every 3-4 months (I do not update it more frequently because all the reuploading and reprocessing is time consuming)

      If you want to join Asirikuy it is also important to do so because of all the things you get besides the data :o) Please watch the introduction video within the Asirikuy main page and the sample video within the joining page to learn more about the site and what its objectives are. Thanks again for your comment,

      Best Regards,

      Daniel

  6. hasegawa says:

    Hello Daniel,
    I came here while researching MT4 historical data.
    You mentioned that you obtained historical data from Alpari.
    Is it different from the data which is downloadable from Alpari-NZ
    historical data center?
    They seems recently provided quality data without too much holes.
    But I found the price is slightly different from the live feed data of Alpari-NZ. It is also diffrenet from the Metaquotes historical Data.
    It is obvious difference when comparing them during year 1999 and 2005.

    What is the data quality of yours? Is it different from the one from Alpari-NZ? I’m taking care of the price not ony holes.

    Thanks and Best Regards,

    hasegawa
    Tokyo/Japan

    • admin says:

      Hi Hasegawa,

      Thank you for your post :o) Our data is NOT the Alpari-NZ data, it is another Alpari data set. Also note that the Alpari-NZ historical data has incoherent GMT shifts because the data is +1/+2 before 2011 and then +2/+3 after this, this can be very detrimental to your simulations. The data we have at Asirikuy is fully +1/+2 GMT and contains very few holes (no important ones that I could find (more than 1 day)). However our data is also different from the Alpari live servers because it is data consolidated from their main feed (so I have been told by my source). Thanks again for visiting!

      Best Regards,

      Daniel

      • hasegawa says:

        Hi Daniel,
        Thanks for reply.
        I checked the data and found you are right about GMT shift .
        Is your data quality best in the world? Especialy before 2005.
        Do you privide your historical data for your members?

        • admin says:

          Hi Hasegawa,

          Thank you for your reply. I wouldn’t say that the data quality is “best in the world” but it is good enough to get accurate simulations in the H1 and upper time frames. You can also get good quality data with constant GMT shifts from forex-historical as well. Asirikuy members get the Alpari data I have as well as a discount for the purchasing of the forex-historical data. Thanks again for posting,

          Best Regards,

          Daniel

Leave a Reply


Fatal error: Call to undefined function mcsp_html() in /home1/danielfp/public_html/mechanicalforex/wp-content/themes/modern-style/comments.php on line 71