One of the main problems we have experienced when using the Metatrader 4 platform is that the quality of the data downloaded from the history center is doubtful. Metaquotes recognizes that this data is “indicative” – which means it is gathered from several different sources – implying that its not as trust-worthy as data which comes from a particular broker feed. The nature of this data makes several problems arise to a various extent on different currency pairs with the EUR/USD being the most “reliable” data set while others like the USD/JPY and NZD/USD have serious data quality problems. On today’s post I will share with you some of the problems of the metaquotes history center downloaded data (the 5 digit broker data set which is the most “reliable” one) and how I finally managed to obtain some truly reliable broker data which will be used from this weekend in Asirikuy.
But I thought you said the data was good ? True, I had performed an analysis of the EUR/USD data from January 2000 to January 2010 using excel and I had found almost no holes in data with only some very small problems attributable to the missing of some small parts of the data (just a few hours through all the set). However during the past few weeks I have been encouraged by a few Asirikuy members – who have brought some problems to my attention – to analyze other currency pairs and the quality of the data from January 2010 till June 2010. The results were quite surprising showing that Metaquotes data has some big and serious issues on this year’s data and particularly on previous years on other currency pairs.
What are these problems ? Well, the main problem we have is that data is missing. The overall quality of the data that is available is good but there are several “batches” where there is no data. There are several BIG problems with this issue as missing data damages the quality of the tests – not only for that period – but for every period that uses this data to derive indicator data. For example, a few days are missing in March on the EUR/USD data, this means that all 14 period ATR calculated data will be different for at least 15 days, simply because the previous data (which is needed to calculate the real ATR value) simply does not exist.
The EUR/USD however is a particularly bad example to point bad data quality because the pair has good quality for almost the entire past 10 years on the Metaquotes 5 digit broker data. However, other currency pairs don’t have the same luck and problems appear as very large batches on previous data. For example, the USD/JPY pair is missing almost the ENTIRE 2004 year (see image above). It therefore becomes a big problem because there could be an important over or underestimation of long term profitability and draw down characteristics depending on the nature of the results for that year. The fact that the systems in Asirikuy use money management that depend on the outcomes of previous positions also makes this quite important since results in the future might be different simply due to the loss or accumulation of profit in previous years. The NZD/USD is another good example of such a case with about three months of data missing in 2008.
After performing a systematic analysis of consistency for the 6-8 pairs we frequently use in backtesting in Asirikuy it became obvious that the quality of this data is not good enough to have reliable backtesting results. Even though Asirikuy systems are robust and the profit and draw down targets might not change a great deal for even a missing year it becomes absolutely clear that the long term analysis of profit and draw down might have been deviated from reality. For this reason I decided to solve this problem once and for all.
The first idea to solve this problem was to buy data for all the currency pairs. Reliable 1 minute data with no holes for a 10 year period costs about 1.5K USD per currency pair meaning that we would need about 5-6K (with some discounts for block purchasing) to get all the data we needed. This option was consulted within an Asirikuy forum thread in which members started to vote whether or not we wanted to make this purchase as a community. However, I started to look for other options as the idea of a community purchase didn’t seem that achievable within Asirikuy (at least in the short term).
Finally I decided to email all the brokers I have “big” accounts with to try to get some files with their historical data. As a client you may have a “right” to data access on your broker so I realized that I had absolutely nothing to lose by asking for 10 year data from all these brokers. My request was answered positively on a few brokers but most offered me only data for the past 2 or 3 years, something which is not good enough to run tests of long term profitability. In the end Alpari UK was the only broker that allowed me to get almost 11 years of data (January 2000 till October 2010). This Alpari data is direct bid data from the broker with NO holes, NO errors and no problems like the metaquotes indicative data. I have run tests on all the currency pairs and I am glad to say that this bid data has amazing data quality, coming straight from a well known broker source.
After receiving this data I promptly converted it to Metatrader 4 format and used it to rerun all Asirikuy backtests. All the systems remain profitable and all the profit and draw down targets are similar although some systems do show better while others show worse results. The above image shows you a comparison of the USD/JPY backtests for Atipaq with metaquotes and Alpari data. You can see that 2004 (the year previously missing) was actually a pretty good year. This weekend I will be launching a few videos regarding this topic (updating backtesting tutorials) as well as the whole set of new backtests. The data will be available for download (in already converted mt4 format for all the different currency pairs) within Asirikuy and the tutorials will explain how to load it to run reliable backtests.
I am happy to say that this will be the last time I will write a post about backtesting reliability. The new data I obtained for the Asirikuy community shows no holes and will allow us to develop systems without any doubt about their results under previous market conditions (provided that they are built in a way that doesn’t exploit the limitations of the back-tester!).
If you would like to learn more about simulations and how you too can develop your own systems that are back/live testing consistency within broker dependency please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach automated trading in general . I hope you enjoyed this article ! :o)