No more MT4 Back-testing! :o) Moving to a professional, C/Python Backtester for the F4 Framework

During the past few months it has become evident – as I expressed in my open letter to Metaquotes – that the Metatrader platform has become a hostile environment for system development and professional live trading. For this reason it has become a priority for me to develop alternative solutions to all the needs that up until now had been fulfilled by this platform. Amongst these needs, one of the most important is clearly the historical evaluation of a strategy’s performance, also known as back-testing, which is fundamental to the development of algorithmic trading systems. During the past two months I have been working tirelessly with an Asirikuy member from Spain (Jorge) to develop a professional and robust solution that would allow us to back-test trading systems outside of the MT4 platform, without all the ridiculous and stubborn limitations imposed by Metaquotes on their strategy tester. Our efforts have created a C/Python tester assembly that is able to perform accurate and powerful historical tests, way beyond the many limitations of the MT4/5 trading platforms. Through this post I want to share with you some of the achievements we’ve made as well as some of the functionality we are planning to implement going forward.

The Metatrader 4 strategy tester has never been an ideal back-testing solution, when working with this tester you always feel like you’re working “around it” and not “with it”. The MT4 back-tester suffers from some severe limitations in functionality that have been generated by the stubborn Metaquotes attitude towards the implementation of trader suggestions. For example you cannot select the spread used for back-testing, when the implementation of such a feature is as obvious and as simple as adding a few code modifications. Besides this there are other problems with configuration, a very limited set of statistics for optimization and a general lack of knowledge about how the different statistical measurements are calculated by the tester. You also have a completely  “hard coded” genetic optimization algorithm that you cannot modify or tweak to match the needs of your particular optimization procedure. There are other factors that inherently affect simulation accuracy, such as the use of constant swap rates through the whole historical period (as I mentioned on a recent post as well). Metatrader 5 fixes some of these things but introduces yet worse problems, such as the inability to load custom historical rates (so you have to carry out simulations with whichever data your broker’s server wants to give you). On top of all of this the MT4 platform has no multi-core capabilities, taking no advantage of modern processor process distribution capabilities.

Screen Shot 2013-06-07 at 4.53.23 PM

We – at Asirikuy – decided to develop a tester with Jorge that would fix all of these limitations, providing us with a very powerful testing framework that could allow us to get rid – for once and for all – of the MT4 platform’s strategy tester. Using the power of the F4 framework it was rather easy for us to develop a tester because we can use the exact same code used for live trading in Mt4, but carry out the back-testing under a completely different environment. There is no uncertainty about the code in simulations and live trading being any different, because the strategy implementations are simply exactly the same (the same code is executed for live trading as it is for back-testing). With all of this in mind we developed a tester that now far exceeds the capabilities of the MT4 back-tester for our back-testing needs. These are the current capabilities of our C/Python backtester solution:

  • Cross-platform compliant. The tester has been coded/developed so that it can be compiled under Linux/MacOS (although we still haven’t run tests under these setups).
  • Multi-core support using OpenMP
  • Supports MPICH, meaning that you can run the tester across even large computer clusters. This is a feature that I haven’t seen in any other back-testing software. Want to run an optimization across 10 computers with quad processors using a total of 40 cores? No problem!
  • Optimizations using either brute-forcing or genetics
  • Genetics are implemented using the powerful open source GAUL library. You can fully tweak ALL the parameters of the genetic optimization (mutations, breeding, migration probabilities, initial population, number of generations, etc).
  • Optimizations can automatically eliminate results with few trades, asymmetric results, etc. This is a huge improvement over MT4 as genetic optimizations do not converge to bad solutions as easily.
  • Wider available statistics for optimization (correlation coefficients, Ulcer index, profit factor, etc)
  • Custom statistics can be implemented. As the source of the tester is fully available, you can simply implement any statistics you want to use that are not added by default.
  • Test systems across multiple symbols. This means that you can carry out an optimization that simultaneously evaluates a system on N currency pairs.
  • Portfolio simulations. Do you want to evaluate how the A+B+C combination of systems would have worked historically? No problem!
  • Accurate historical swaps! Derived from historical interest rates obtained from Oanda.
  • Ability to select any spread level desired :o)
  • Images generated for balance curves, historical swaps and volumes.
  • HTML/XML and CSV format reports are generated after each test. The HTML reports even allow you to sort trading results by using the column headers!
  • Console and visual interface access for easy configuration and running.
  • Configuration files are easy to share, meaning that you can allow other people to easily reproduce your tests by simply sharing a file. No more irreproducibility!

As you can see our tester implements capabilities that far exceed those of the MT4 platform, giving us the opportunity to completely get rid of the MT4 strategy tester (as we have confirmed as well that our simulation results are accurate as well). With our new back-tester we can now perform tests in the manner that WE choose to be the best, optimize the variables WE want to optimize in the way WE want to do it and evaluate the setups WE want to trade. We can now run our portfolios and see how they have done historically (in a direct manner) and we can also carry out optimizations that target strategies that match our stability criteria. We can also simultaneously optimize a strategy across several instruments or even different feeds of the same instrument, to find parameters that generate the most robustness or the least broker dependency.

Our implementation is however quite new (only about 2 months from our first code writing) and we still want to implement other things to further the capabilities of our testing platform. For example next week I am going to be working on the implementation of multi-symbol capabilities, giving us the ability to test a system that uses information from several different symbols to make trading decisions. This will give us the possibility to develop systems that exploit inefficiencies that work through the whole currency market, not focusing on information from a single symbol to make trading decisions (things such as correlation based systems could be properly evaluated with this implementation).

Screen Shot 2013-06-14 at 6.13.39 PM

As you can imagine, the above gives us a potentially huge advantage against traders who only have access to the MT4 platform, simply because we have the capabilities to work above the limitations of this tester. We can run tests in a faster way (take advantage of multi-core or computer cluster setups) and we can also carry out simulations that the people using MT4 simply cannot do (for example optimizing a system simultaneously for various pairs). In addition the added accuracy of our simulations – thanks to our accurate historical spreads – gives us an additional edge as we cannot be “fooled” by problems that stem from the assumption of constant historical swap rates (as discussed in my last post). We can also run directed optimizations – as we can control the variables of the genetics and target values – meaning that we can arrive to better solutions within the parameter space in a faster manner.

Last week I carried out what I hope will be my last back-test ever in the MT4 platform. Right now I am using our C/Python tester and I am extremely happy with it’s speed, results and capabilities. From now on I will only run my tests on this implementation and hopefully by the end of this year – as I mentioned in my open letter to Metaquotes – I will be able to get rid of MT4 for live trading as well. I will also start to move our community members away from MT4 back-testing in the hope that they will embrace our new and powerful C/Python back-testing solution :o)

If you would like to learn more about back-testing outside of the limitation of the MT4 platform please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

 

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

30 Responses to “No more MT4 Back-testing! :o) Moving to a professional, C/Python Backtester for the F4 Framework”

  1. Odkhuu says:

    Woow , you made very good effort , i appreciate it, happy trading!

  2. Franco says:

    Excellent Daniel, this is a milestone for this community. Looking forward to work with the Asirikuy Tester.

  3. Dz says:

    !!!!!!!!! great !!!!!!!

    Just thinking what about feed data used today? Technically its from Alpari & ForexHistorical and may have some relation to MT platform? Does you ever looked recently on other data feeds for compare in this new tester? As I understand FXCM via API has ticks with exact spreads and maybe OANDA via API also…

    It is interesting to be able to test strategy easily via several datasets, even if thess API feeds have only last 5 years or smth. because it theoretically should not have MT execution problems.

    Other interesting thing will be to have performance metrics for different brokers. Take API feed, take live results for a year, do backtest for this year and have statistical comparisons btw live & backtest. This way we could be able to quantify broker execution performance as some standard step in trading or tradability of the strategy.

  4. Rodney says:

    Daniel, Jorge and yourself have done what we all wanted Metatrader to be able to do in just two months! Shame on Metaquotes, especially with their attitude towards using the last recorded spread for backtests and also MT4’s lack of multi-currency backtesting. The latter of these two problems has driven me crazy (although Asirikuy had some really nice solutions thanks to Gabor). I am fully in agreement with you in your move away from MT4. The empowerment this new tester will give Asirikuy members is going to make trading much more exciting and rewarding.
    Regards
    Rodney

  5. Sim says:

    Programming + Forex trading = Mechanical trading in forex. I just stumble on this website, which i think is awesome. Looking forward to work with you guys in future.

    I think that this type of approach of forex trading is the Holy Grail of forex’s trading method. Nice!

  6. Jim says:

    This is an interesting development, Daniel!
    It would be interesting to see a comparison between the equity charts & statistics from a given EA for both platforms.

    Jim

  7. Maxim says:

    Daniel,

    A really great work and a huge achievement!
    It will be interesting to know whether there is live/back compatibility for strategies, running live few years (like Ruphay), with backtest results for the same period of time.

    Maxim

  8. Sebastian says:

    Wow! I´m impressed to say the least!
    Hats off to Jorge and you Daniel, this is a great milestone!

  9. Martin says:

    Hi, so you are able to backteSt any given EAs that was developed for mt4 with your tester?

    • admin says:

      Hi Martin,

      Thank you for your post :o) No, the tester is only for experts developed using the F4 framework (our programming framework at Asirikuy). Our framework is designed in ANSI C and the systems can be executed in MT4/5, Oanda, Dukascopy (these last two currently being implemented but already functional — see some later blogposts). We designed our framework to allow us to eventually move outside of MT4/5, which we are preparing to do soon! I hope this helps,

      Best Regards,

      Daniel

      • Martin says:

        Hi Daniel,

        first of all your work is well appreciated, and maybe you get it to a quality were it can directly compete with the MT4 backtester to finally have some competition going on here.

        Unfortuanetly we have our EA’s coded in MT4 and they are running on live accounts. But I am sure they can be translated to your framework by a good programmer, so we’ll discuss that internally here.

        Anyway I have a question regarding your tester. Are you able to do backtest with 99,9% tickdata + historical spreads?

        What are roughly the passes per minute compared to MT4 if using the same processor? Right now we have 2 computers running 24/7 with 16 MT4 instances doing optimization runs. As we are running each MT4 application on one real thread/core I am wondering if we would have a higher throughput with your tester?

        You can also just e-mail me.

        best regards
        Martin

        • admin says:

          Hi Martin,

          Thank you for your comment :o) Since our systems are developed to work only with data from past closed bars and profit/loss targets greater than 5-10X the spread, we have mainly implemented our tester to simulate this type of systems accurately. This means that currently our back-tests aren’t executed on a “tick-by-tick” manner, as this is unnecessary for our systems (it wouldn’t mean any increase in simulation accuracy). Our simulations are completely accurate for this type of strategy but inadequate for scalpers, systems with low P/L ratios or systems that use information from the currently open bar (needing tick-by-tick execution). It is also worth mentioning that we will probably not be implementing such a functionality in the short/medium term as we are not interested in the development of scalping systems or systems with very low P/L targets. I hope this answers your question :o)

          Best Regards,

          Daniel

          • Paul says:

            Hi Daniel

            Regarding your response to Martin in which you mentioned that tick-by-tick evaluation in your back-tester is not likely to be implemented in the future…

            Should I have need to further develop the back-tester in order to accommodate tick-by-tick evaluation, is the code open as such that I (or my developer) might make that change ourselves?

            Also, can you comment on the speed of the underlying back-testing code compared to MT4 back-testing. That is – if it were testing tick-by-tick, single thread to thread comparison with MT4 (very roughly) what would you estimate?
            Your backtester is fundamentally written in C interfaced via Python – is that correct? (Which would imply its likely to be running about as fast as its possible to go)…

            Thanks!

          • admin says:

            Hi Paul,

            Thanks for writing :o) We implemented tick-by-tick testing some time ago. The code is in fact completely open for Asirikuy members so you are able to see how this functionality is implemented and change anything if you require. As you point out the back-tester is implemented in C interfaced via python so testing is much faster than in MT4 although the exact difference in speed will depend on how complex the calculations you carry out on each tick are. A one year tick test using something like a simple MA calculation should take just a few minutes. Let me know if you have any additional questions,

            Best regards,

            Daniel

  10. reptile says:

    Why you not use tradelink or OEC platform with C#? What is your opinion?
    Exist xls ala resolver for python.. maybe you should look to join/include your part of work.. can be something new for GUI tp/sl etc..
    We still building tools not main code for target.. thats mind we loosing time.

  11. Jon says:

    the main problem with implementing your own platform is that you must then find or develop a bridge (to the broker api) for this. So it might be great to validate code and test strategies, but to migrate to a live environement requires the API. Many brokers make API access inaccessible unless the starting capital is >$100,000.

  12. Leon says:

    Hi, I’m busy coding various EAs (MT4) that’s to be used on a multi currency trading system
    Simply put, I will run 2 larger grids (EURGBP and USDCAD) and smaller high frequency grids (same pairs) together on a single account.
    So it’ll be 4 charts USDCAD-M1, USDCAD-M5, EURGBP-M1, EURGBP-M5, each with it’s own EA attached, on a single account.

    Can your software backtest this combination of 4 EAs on 4 charts, from a single account?

    Thanks!

    • admin says:

      Hi Leon,

      Yes, our software can in principle run this back-test (As you post it),

      Best Regards,

      Daniel

  13. david says:

    Hello,

    All points you quote are expected.
    How a generic software could be so flexible and adapted as a custom software?
    All big banks and funds builds own tools. I know it since software building is my job.
    The problem with building our own backtesting software is that it’s a hard and long job. It can last one year or more to have the expected results.
    For a real trader who seems restrained by MT4 or another tool like MetaStock it’s the way. For casual traders, the investment is too expensive.

    Good luck :)

  14. dan says:

    Can I use for testing short term scalping?

    • admin says:

      Yes, the framework can now back-test using tick data. However beware that differences between tick data from different brokers can be very significant and slippage can be very important for scalping systems so the real accuracy of the simulations will depend on whether you have the exact same tick feed for live trading/simulations and whether there is important slippage on your broker.

      • dan says:

        awesome! What tick data do you offer? I plan on using Oanda broker.

        • admin says:

          We have 1M data for several symbols back to 1986 but we don’t offer any tick data. However you can download tick data from Dukascopy for free or you can actually download tick data directly from Oanda from 2004 if you have a deposit of over 1500 USD. If you plan on using Oanda you’ll also be able to trade with them directly using our Asirikuy Trader program which communicates directly with the Oanda broker API.

  15. Mohamed says:

    Hi can this system take entry and exit signals from 1 instrument and apply them on another instrument to run a backtest on MQL4? Do any systems of such sort exist?

    • admin says:

      Hi Mohamed,

      Our framework allows you to back-test systems as you describe, however these are back-tested using our own proprietary back-tester (not MT4) as mentioned on the post. Let me know if you have other questions,

      Best regards,

      Daniel

  16. Raghbir Gill says:

    I have a few MT4 EA’s i would like to test.Can your tester test multi pair currencies simultaneously and faster using the metaquotes indicators and EA’s ?

    Please advise.Thank You

    • admin says:

      Thanks for writing. Our tester does support multi-currency tests but our tester is not compatible with MQL4 files. To use our tester you will need to recode your system and indicators in our F4 programming framework.

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons