Are You Backtesting Correctly ? : Six Common Technical Mistake That Will Make Your Simulations Useless

When simulating the performance of a trading strategy using historical data within the Metatrader 4 platform there are many things that you can do which will inevitably end in bad performance and unreliable results. Many of the reasons why people regard backtesting using Metatrader as unpredictable and difficult to reproduce are a consequence of one or several technical problems which can arise due to the lack of carefulness of the trader running the evaluation. Knowing about these problems and taking action so that they do not affect simulations is something necessary to arrive at reproducible and reliable results. I have proved – within several trading strategies – that simulations can be reliable and easily reproduced if special care is taken to avoid technical pitfalls.

Within the following paragraphs I will share with you the six main technical reasons why traders arrive at unreliable simulation results that greatly over or under estimate the profitability of their trading systems. These problems can be easily avoided through some simple measures that can ensure that simulations are as reliable and useful as possible. Falling into just one of these problems can cause back-testing results to be utterly meaningless so avoiding them is of primordial importance for anyone interested in the accurate evaluation of trading systems and expert advisors. These are the technical problems you might encounter :

1. Using a 4 digit broker to run simulations. Many people think that the most accurate simulation results for their systems are obtained when running backtests with their broker’s Metatrader 4 platform. However they do not realize that backtesting data is ALWAYS downloaded from metaquotes servers and that the 4 digit broker data set downloaded from Metaquotes contains MANY errors which make simulations totally unreliable. There are major gaps in price in the lower time frames, many daily candles missing large segments of volume, introduction of Sunday daily candles on some years, etc. If you want your simulations to be reliable you need to use and ONLY use five digit brokers for backtesting which download the much more reliable five digit data set from metaquotes.

2. You are using the weekend spread. Another very common technical problem people come across is the running of simulations on the weekend when the spread is extremely high in some cases. When you run a simulation during the weekend using this spread values you will have much worse results than what you would have when using the regular spreads provided during the trading hours of the week. In the end you should always perform your backtests in trading hours or change the spread within the Metatrader 4 platform (we use a script in Asirikuy in order to achieve this).

3. Your strategy trades below the 1 hour chart. During the past few years I have tested and run live/back testing consistency analysis of several strategies that run within the 30 min, 15 min and 5 min time frames only to find out that their results are each and every time inconsistent with simulations. The reason why this is the case is because the lower the time frame the more prominent the effect of one minute interpolation errors when determining things such as indicator values becomes. The broker dependency also increases exponentially and when trading 5 minute charts it becomes so high that the simulations are utterly meaningless. The fact is that variability caused by broker dependency and interpolation errors within these time frames is SO high that you can have totally different results between your backtests and reality. The problem is less pronounced for the 15 minute chart and only a small effect occurs on the 30 minute chart but the problem is not almost completely eliminated until you move to at least 1 hour charts.

4. Your Take Profit and Stop Loss values are within 10 times the spread of the instrument you are trading. When you are running simulations of systems that use these type of trading obtaining reliable results is impossible, not only due to the problem with one minute interpolation errors (which for this case is huge) but because of execution variables (such as re-quotes and spread widening) which prove to be VITAL in the actual real-life profit of these strategies. If you want your simulations to mean something and provide you with some approximation to valid profit and draw down targets then your average Take Profit and Stop Loss must be above 10 times the spread.

5. You are not recalculating your data before each backtesting run. Something which is extremely important is the recalculation of data before starting each new simulation. When you load a chart or when your demo feed sends a tick to your platform there are sometimes history recalculations which corrupt your data and cause your simulations to become erratic and invalid. In order to correct this problem you must recalculate your data within the history center before running every back-test. This can be achieved by going to the one minute section of the instrument you want to recalculate within the history center and clicking the download button until it prompts you to recalculate data. Doing this ensures that your data will not suffer from corruption from your demo feed.

6. You are running a backtest over the last 1-3 months. Your historical data is composed of the data you download from Metaquotes servers and the data you obtain from your live/demo feed from your broker. The last 3 months of testing data are usually downloaded from your broker while the data before pertains to the history center. Usually if there is a time stamp mismatch between your platform’s live feed and the Metaquotes data there will be massive generation of errors within the past 3 months of data as the program gets “confused” from these differences. If a chart of the instrument you want to trade shows massive gaps after you do a historical data recalculation then this is a problem. You can generally avoid this by only running backtesting that end three months before the current time.

Certainly the metatrader 4 platform has many limitations and the above restrictions limit us to the development of certain kinds of trading strategies. However this doesn’t mean that simulations are unreliable but mainly that great care has to be taken in order to make the backtests reliable, reproducible and coherent with live trading results. By following all the above suggestions and avoiding this technical pitfalls you will be able to obtain reliable backtests of your trading strategies which will allow you to get a good picture of the possible long term performance of your trading strategies.

If you would like to gain a true education around automated trading systems and how you too can design strategies that achieve reliable simulations with accurate profit and draw down targets please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly
You can leave a response, or trackback from your own site.

11 Responses to “Are You Backtesting Correctly ? : Six Common Technical Mistake That Will Make Your Simulations Useless”

  1. townjet says:

    Thanks for the post Daniel.
    I needed this post. I knew about most of the 6 items you mentioned by trial and error, but I did not know about how incorrect the 4 digit can be and that the last 3 months can have gaps (Though I suspected something was up when I was seeing those gaps :))

    Thanks

    JT

  2. Tcxmon says:

    Daniel-

    Thanks for the post.

    I've had great success testing the God's Gift EA's since you re-released the new versions earlier this week. I was able to reproduce your back tests and get excellent results and smooth-as-a-babys-bottom equity curves on the 10-year backtests for the God's Gift EA's.

    Also, you are 100% correct about Alpari having the best back-testing data. I was able to run 10-year back test with zero chart mismatch errors!

    On Saturday morning, I had a lot of issues backtesting with Alpari due to the weekend spread. I tried to adjust the spread via the symbols.sel file (over a few hours) with no success. In each case, I clicked the "Symbol Properties" button to check the spread to find my changes not taking.

    I went back and looked at the other symbols.sel files on the system and found the dates on those files set some time in the past, much further back than the last date used by the platform.

    So I tried to delete all the symbols.sel files from the MT4 folder and still found MT4 not to be writing a new symbols.sel and instead getting the spread from some other source. So I have yet to fully understand from where MT4 is getting the spread info.

    In any case, i'm sure we will figure it out. I do appreciate all the time and effort you spend helping us to figure out the complexities of MT4 and Forex.

    I've e-mailed some EA vendors just to get a 2-line, unhelpful response.

    So thanks again for all you do and i'll share any further insights I find on the source of the spread info.

    Thanks,

    Chris

  3. Daniel says:

    Hello Chris and JT,

    Thank you very much again for your comments :o) I am very glad that you are such faithful readers !

    @JT : Yes, I was also surprised to find about all the problems with the 4 digit data. Actually it only became clear after I did a very thorough analysis after developing Watukushay No.2. The issues with the last 3 months of data became evident after I started doing back/live testing consistency tests last year. I am glad I can share this knowledge with you so that you can improve your backtesting skills :o).

    @Chris : I am glad that this new EA version is now working better for you ! :o) This new version has a lot of improvements with a massive reprogramming of the code (without changing the logic), a whole new user interface and adequate error handling capabilities. I took special care in making sure that backtesting was reliable and reproducible so the problems with simulations for this EA should be entirely solved :o).

    Regarding the spread modifications, I did receive a few emails from other Asirikuy members facing similar problems reason why tomorrow I will be releasing a re-make of the spread changing video tutorial to avoid the problems you are currently having. I have tested the new procedure on several computers so you should be able to solve your problems using it. The video should be released tomorrow before 12 am EST along with the other videos and the weekly text newsletter. Of course if you have some problems with changing the spreads after you follow this new procedure feel free to share your experience within the Asirikuy community forum so that me and other members can help you solve it.

    Thank you very much again guys for your comments and visits :o)

    Best Regards,

    Daniel

  4. Gabor says:

    Hello guys,

    I follow the follwing steps when I run backtests:
    – delete all the *.hst files in the /history folder before I start the terminal
    – start the terminal, log in and download the M1 data from metaquotes in history center
    log off the terminal (I mistype the password and login), this way the terminal gets offline and can't update the feed and the symbol.sel (which contains the spread and other trading environment variables), in addition you don't need to recalculate data after every backtest run (point 5). This however doesn't solve the problem in point 6, so you have to run backtesting that end three months before the current time
    – close terminal, copy the hacked symbol.sel into /history folder and then re-open terminal
    – running backtests :)

  5. Jon says:

    All of the problems you list in this article are solved by using tick data to ‘replay’ (simulate) your strategies. In lieu of tick data, 2000 < millisecond data would be sufficient. Brokers like Interactive Brokers provide a few years of 400ms data for free. Hey, its free. 400ms snapshots (5 price updates every 2 seconds) are accurate enough to simulate most non-HFT strategies, and you can acheive the highest modeling quality with tick-by-tick data.

    The problem is that metatrader 4 does not support tick data (or seconds/ms bars) importing out of the box. The workaround is to use Birt's Tick Data Suite, TickStory, or some other 'hack' to import tick data. The only other option is to port the strategy into another charting platform and run the backtest that way. This is technically possible, but it is always easier to test in the same algorithmic environment that you will trade live with.

    Metatrader 5 does not support ANY 3rd party data. I'm surprised that MetaQuotes is being so stubborn about supporting tick data, seeing that they are promoting ALGORITHMIC trading.

    Correctly simulating the proposed strategies with tick data (real spreads, volume fully simulated) will shave off years of guesswork and will improve the accuracy of trading strategies. The real problem with backtesting is over/under-optimization and not acknowledging the weak points in the strategy and developing workarounds. I only wish I knew about the possibilities earlier in mt4 (tick data backtesting).

    • admin says:

      You are forgetting about broker-dependency. Some strategies can give very different results even with tick-data just because the feeds are different between your tick data provider and your real live data. Some strategies do not give trust-worthy simulations, even if you use tick level data.

      Also please avoid placing a URL when leaving comments, the URL is automatically removed when your comment is processed,

      Daniel

  6. Forexmm says:

    Hi Daniel,

    Loved this article. I have quite a few questions for you if you have some free time. My email adress is attached with this message.

    – forexmm

  7. Forexmm says:

    Is number 3 of your article still a valid point even if you get tick data to run your expert adviser through the strategy tester?(what jon said in a previous comment) Reason I am asking is because my EA is ran on m1 and have been running my program through m1 strategy tester.

    I am using a multi time frame EA that I created that I cannot seem to get good results from using the strategy tester yet I get good results running the same settings live. It uses m1 m5 and m15 charts to produce long and short signals.

    Am I doing something wrong? From my understanding meta trader 4 (only meta trader 5 can accurately test multi time frame signal EAs) cannot test multi time frame EAs accurately.

    Is there a way to test multi time frame EAs accurately?

    Could you shed some light on this issue I am having Daniel?

    – Forexmm

    • admin says:

      Hi Forexmm,

      Thanks for writing. Simulation accuracy depends on several things, including how your system is coded, how much it depends on broker data structure, how much it depends on the current spread, etc. For example you might get inaccurate results simply because your back-testing data is GMT 0 and your broker has data in GMT +3, meaning that your broker has different weekly starting/ending times and therefore totally different candle structure.

      Metatrader 4 is also not a viable platform for back-testing. It is like a black box where you know very little about how the simulation process is actually executed. This is the reason why we had to code a separate back-tester at Asirikuy, which we know exactly how it works and therefore can access rates and carry out simulations in a manner that is predictable and as accurate as possible for the level of data resolution used. Our tester uses systems coded in the F4 framework so it does not work with MQL4 systems. Simulating a multi-timeframe EA accurately in MT4 is possible but a tedious task. You need to get the entire data arrays you will use each tick in your system, then make sure to refactor it so that the candle structure matches your live broker exactly (same DST, GMT shift, weekly opening/ending times). You can then simulate and compare with a live trading result to see if behavior matches.

      If your system depends on tick data granularity then it is most likely that using data from a different broker will also show different results, regardless of how you simulate because your system is too dependent on the fine grain of the data and broker feeds are always different. In this sense it is important to simulate on the exact same broker data that is used for live trading (historical tick data from the exact same feed).

      Overall I would recommend that you change to something different than MT4 if you’re serious about algorithmic system development. No professional that I know uses this platform. I would also recommend you choose to code your systems such that accurate simulations are less feed dependent and less dependent on having high resolution data (for example using information only from closed candles and avoiding using the Bid/Ask values). Using SL/TP values significantly larger than the average spread (5-10x) will also reduce this issue. Thanks again for writing,

      Best Regards,

      Daniel

  8. Forexmm says:

    Thank you for the reply.

    So basically mt4 cannot access multiple time frames. I just received an email from metaquotes and they said it is not possible.

    They did however say it is possible in mt5.

    Buy Asirikuy? I wish they had a demo version/trial version.

    Are there any other free alternatives that you recommend that can upload mq4 expert advisers into and test with accurate results? Specifically one that can use an expert adviser that accesses multiple time frames to confirm buy and sell entry points?

    – Forexmm

    • admin says:

      Hi Forexmm,

      Thanks for writing. The problem is basically that you’re using a platform with some innate limitations that will make it difficult for you to get accurate results. It is simply necessary for you to migrate to a more professional platform that allows you to perform what you want. If you want to go the free route you can code a tester from scratch in python/R (there are several online sources that may help you with that task) and recode your system so that it can be used with your tester. However how worth it this is depends on how much you value your time (if it’s worth it for you to spend this time coding) and how fast you can code such an implementation. Thanks again for writing,

      Best Regards,

      Daniel

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons