The Metatrader 5 Series : When Huge Backtesting Differences Appear

If you look at my last few weeks of posts I have been very excited about the new qualities of the Metatrader 5 trading platform and the benefits it brings to system development regarding faster execution, faster optimization and added flexibility. Using my prototype implementation of Watukushay FE I analyzed trading over several different instruments and I finally came up with the “starting point” of intra-currency trading for this well-known freely available trading machine. This week I decided to do a small experiment and compare my results of Watukushay FE for the AUD/USD on Metatrader 5 with those found out with Metatrader 4 and the results found were extremely surprising. On today’s post I want to talk about my findings, the possible causes of the issues found and what I will be doing to investigate the nature of this problem and what solutions can be implemented to deal with it.
During the first post comparing Metatrader 4 with 5 and the backtesting results of Watukushay FE we already saw a small yet noticeable difference between the testing results obtained on both trading platforms. I talked about the possibility of these errors being feed-related and the fact that the Metatrader 5 history feed might be more reliable since it has been “remastered and fixed” for this new platform. However the difference was small and therefore there was no substantial issue besides an anecdotical note pointing out this curious fact.

However when I decided to run the initial AUD/USD tests on Metatrader 4 to compare the results I obtained with Metatrader 5 the difference changed from “noticeable” to “abismal”. The pictures below show you the results for MT5 and MT4 using the exact same settings on the AUD/USD currency pair backtest from 2000 to 2010. The overall equity curve is very different and the results do point out that something is substantially very changed between the historical feeds of MT4 and MT5. I first thought that the difference would be due to the presence of Sunday candles but this turned out to be false since the MT5 feed doesn’t have any of them, so regarding this aspect it is the same as MT4. I then thought about the possibility that the whole difference is caused by important changes in data prior to 2006 (before metatrader 4 was launched) and the fact is that data differences are NOT limited to pre 2006 periods, the whole historical feed is different between both trading stations and meaningful differences are present. If you analyze the results you’ll notice that almost all candles have different – if only very slightly – high/low/open/close values pointing out that RSI and ATR values will be very different. The change in one minute interpolation mechanisms is also not likely a factor here as Watukushay FE strictly controls bar opening on both its MQL4 and MQL5 implementations.

What causes such a dramatic change in profitability ? To get to the bottom of this problem I decided to strip down the logic to its simplest form and eliminate the closing logic of the EA, leaving only the entry rules. This shows us that there is still some difference between backtesting results (shown below). This means that differences in results are caused by differences in the RSI and ATR indicator calculations which are dependent on each backtest’s particular historical feed. Stripping down the logic does reveal that most dependency is located before 2002 with results beyond this date being in better agreement. However there is still some dependency which is caused by differences in data between both historical sets beyond this period.

Since we simply cannot know for sure which of the two historical data selections is better – and they are probably both valid within normal broker differences (with the 2000-2002 data being very different probably due to differences between feeds in this period) – it becomes a wise decision to run backtests on both and trust the less profitable results to calculate profit and draw down targets. In some cases like the EUR/USD backtests this proves to be trivial but on others like the one I showed you today doing this mixed analysis proves to be extremely important. I will email the people at metaquotes to get some information about the different nature of the feeds and I will let you know once I have more information about their origin. However up until now all backtests of Watukushay FE seem to be more profitable on MT5 (meaning that our MT4 simulations are in fact the worst case scenario). Investigating other issues which may be related with the closing mechanism of orders in MT5 is also something I am currenlty doing since I have seen that the differences when the closing logic is enabled seem to have other strong causes besides simple feed dependency (more on this on a later post !).

If you would like to learn more about automated trading, the evaluation of expert advisors and the programming of your own strategies please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !

Print Friendly
You can leave a response, or trackback from your own site.

7 Responses to “The Metatrader 5 Series : When Huge Backtesting Differences Appear”

  1. Alex says:

    My guess is the spread. I have done some tests myself with my own EA's and it appears that for only the last year or so of data is a spread returned that is greater than 0. On old data the spread reads 0, which is of course a big problem. Im hoping they are planning on rectifying this soon.

    Nice blog btw :)

  2. Daniel says:

    Hi Alex,

    Thank you for your comment :o) I would tend to disagree with your conclusion since the TP and SL values of Watukushay FE are much greater than the spread (20-30x) so having a spread or not having one would show more profitable results but the shape of equity curves would still be similar since globally the same profits/loses would happen. However from 2000-2002 the equity curves are almost opposite in MT4 and MT5 and an analysis of the entries and exits show that they are very different so in fact the spread is most likely not the issue here. However – as you say – hopefully they'll fix this spread issue soon if it is still there ! Thanks again for your post and nice words about the blog :o)

    Best Regards,

    Daniel

  3. Alex says:

    Hi Daniel,

    My pleasure, always great to read up on some good ideas.

    Yeah I see what you mean about the 2000-2002 area, there is definately something else going on. I shall keep my eyes open for differences too as i'm working predominantly with mql5 now.

    Just wanted to let everone know about the spread though, can seriously skew results and optimisations unfortunately :(

    Regards,
    Alex

  4. Francisco FX says:

    Hi, Daniel, Francisco here. Based on the nature of the issue experienced with WFE, there are only two possible explanations IMHO:
    1. Like you mentioned, a difference in the historical data quality. About this matter, I have recently downloaded MetaQuotes data on MT4, and even though I still get 90% modelling quality, I have holes going all the way to 2010 included (believe it or not). That can affect backtesting results dramatically, in my experience;
    2. Some sort of coding error from when you migrated the EA from MQ4 to MQ5 language. In order to verify this, I guess you would have to look at the detailed trading graph, see if the trades follow the EA trading logic.
    Those are my 2 cents, ;-)

  5. Daniel says:

    Hello Francisco,

    Thank you for your input :o) The errors you mention on MT4 data appear only on the 4 digit dataset reason why only five digit platforms should be used to backtest systems. They could also appear on five digit data due to demo feed corruption which is quite common. A lot of care is needed to avoid these pitfalls but I can confirm that my metaquotes MT4 five-digit data contains NO holes whatsoever.

    Regarding coding errors, this is something I am currently looking into but entries seem to line up with the logic perfectly on both cases. Maybe a slightly different implementation of the indicators could also be a cause ? I will investigate this further and write a post about it.

    Thanks again for your comment,

    Best Regards,

    Daniel

  6. Francisco FX says:

    Hi Daniel,
    Thanks for getting back to me so quickly.
    To be honest, I'm getting the holes on different brokers (even on 5 decimal ones, like Alpari), after several hst file deletions (prior to downloading the data from MQ Inc.).
    About the issue, you seem to be a very methodical person, so I'm sure you'll be able to find the cause of the discrepancy. Let me know if there's anything I can give you a hand with.
    Best of lucks!

  7. Daniel says:

    Hi Francisco,

    Thank you for your comment :o) Make sure you send me an email with a picture of some of these holes to my email at ekans_ at hotmail.com so that I can compare with my data. I did a total search of errors in my data using excel and I found no problems. However I am obviously always in the lookout for any so I'll be glad if we could get to the bottom of this problem you're finding.

    I will certainly search for the answer to this question with as much depth as possible and I'll write a blogpost when I have enough information. Thank you very much again for your comments :o)

    Best Regards,

    Daniel

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons