The Order Wrapping Nightmare: Building an NFA-compliant Order Manager Library for MT4

This morning – while working on the F4 framework with Morgan – I was reminded of a project that I had stopped working on about a month ago. This project – that had become very frustrating – was the development of an order wrapper library for the Metatrader 4 platform in order to allow for the virtual taking of multiple orders while remaining in an NFA compliant environment  Essentially what I want to achieve with this is to create a library in which positions can be taken in any way that is required while keeping only one net position opened per symbol. This way you can trade your systems in whatever way you wish while not violating the limitations imposed by NFA brokers. On today’s post I want to discuss why I found this problem so challenging and what solutions I will be trying soon in order to circumvent the issues found.

My original idea to create an NFA compliant virtual order manager for MT4 centered on the use of an external database which I used to store order information. This seemed like the best idea since you want to have a place where you can have organized virtual order information in a way that is easily accessible and which allows you to carry out modifications without any problems. The idea was quite simple, you wrap all the relevant MT4 order sending and querying commands – like OrderSend(), OrderSelect(), etc – so that they get redirected through a library that performs the appropriate task. When a system opens a position the library creates an entry in the database and then the library opens/closes a given lot size on the requested symbol to match the required net positioning approach. So if a strategy opens up a long for 0.01 and another opens up a short for 0.02 lots you will have two virtual orders in the database – a long for 0.01 and a short for 0.02 – but in reality you will only be short 0.01 lots on the EUR/USD. Perfect (?).

Well, not quite. While I was programming this implementation it soon became apparent that things are not as easy as they appear because you have two asynchronous places were trades are being taken. There is a virtual order-book where trades are “taken” and there is the real trading scenario where trades are really taken. The problem is that these two should match all the time, otherwise you will run into some very nasty problems. For example I realized that you should always open virtual orders first because if for some reason the net positioning part of the wrapper fails to open up the order (a crash in the middle for example) it will be obvious that the trade wasn’t opened properly because the open position will not match the sum of the lot size of the opened virtual positions. You can then open whatever is needed to correct the positioning — although obviously the price you get will not be the same you intended. The same applies to exits.

However my problems with the database engine started to become bad when I started to put the implementation to the test. When two or three instances were running there were database access problems because sometimes they all wanted to write to the DB at the same time. There were also problems related with the synchronization of trades when two opposing trades were opened at the same time because sometimes the synchronization reacted in the middle of the process and several modifications to the net positioning were done. For example if a long for 0.05 lots and a short for 0.05 lots were opened at the same time, the synchronization engine sometimes opened a long and then closed it, incurring in an unnecessary trading cost. Additionally it soon became clear that the cost of a DLL cost and DB access would be problematic when doing significant amounts of testing since the code added significant overhead to the computational cost of running the system implementations. The delay between virtual order recording and the synchronization of the net positioning also meant significant slippage, something which was simply not acceptable.

I was ready to give up the idea of an MT4 order wrapper but we then started to discuss with Morgan the fact that we will need a wrapper anyway when we build the front-end for MT5. Since MT5 uses a natural net positioning approach, it is obvious that if you want to trade several systems you will need someway to manage your open orders. Then I realized that if this is something many people will need, someone had probably already done it. After some simple google searching I then found the MQL5 Virtual Order Manager, which  implements the exact same concept I wanted to implement on MT4 (but for MT5). The approach taken here is quite different and the results are quite good, this wrapper – as far as I could see – is able to do everything I wanted to do without the need for an external DB or DLL calls. The great thing is that – with this implementation already available – the building of the MT5 front-end will most likely be a breeze. 

The approach used by this coder was actually quite simple, using global variables inside the platform to store order information. Although this is something I dreaded – because global variable use seems somehow disorganized when you can use a DB – the approach actually faces much less problems than my far more sophisticated (and yet worthless) approach using an external DLL and a proper DB implementation. Although the approach in MT5 is much simpler by nature – because net positioning naturally shifts from long to short – I can see the idea implemented in MT4 without any major problems. This will be much faster than using a DLL and will allow for much less problems dealing with synchronization issues.

My next try for the implementation of the MT4 NFA compliant order wrapper will now be much simpler, trying to keep things to the MT4 platform in order to wrap all order functions in the smoothest possible way. I guess you guys will have a ton of suggestions for the implementation of the Order wrapper, because many of you are excellent coders :o) Feel free to leave a comment with your opinion and any advice you think will be useful in the coding of this library. Hopefully within the next few weeks I will have a shiny implementation to show to the community :o)

If you would like to learn more about trading systems and how you too can learn to code and analyse your own trading strategies    please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly
You can leave a response, or trackback from your own site.

5 Responses to “The Order Wrapping Nightmare: Building an NFA-compliant Order Manager Library for MT4”

  1. Maxim says:

    Hello Daniel,

    Using global variable assumes that you use one platform for one account. Although this is true most of the time, a special care must be taken when using virtual order manager. Additionally, what will happen if to instances want to update the same global variable simultaneously?
    I would propose to use a queuing mechanism to achieve what we want:
    -The trade is updated by the trading server (aka OrderXXX function succeeds).
    -The notification is sent asynchronously to the virtual order manager.
    -The virtual order manager updates the storage being it global variable or database.

    Please let me know what you think,
    Maxim

    • admin says:

      Hi Maxim,

      Thank you for your comment :o) This is exactly what I tried as this was also your initial suggestion. However I faced the problems I highlighted in the article, when you have a virtual order manager that updates storage asynchronously you get into a ton of slippage problems because the time it takes to update can be too long (you also get tons of re-quotes). From what I have seen the synchronization and saving of the virtual trade need to happen within the same thread, otherwise execution can be very bad. There are also problems when you trade a symbol and your asynchronous manager is running on another because syncing of rates becomes a problem as well due to ticks between symbols not matching in time. From this I learned that it is absolutely mandatory for the systems to call the library and for the library to execute all commands with the system thread (no asynchronous manager should exist). Let me know if you have any other ideas :o)

      Best regards,

      Daniel

  2. Fd says:

    Hi Daniel,

    from the problems you described I think following should work:
    – there must be 1 single thread which does all the order communication with the broker
    – for each instance a virtual transaction log needs to be kept by the order thread; from the transaction log a virtual order table per instance can be constructed
    – each instance can query the order thread about it’s own current virtual order table
    – the order thread has the task to keep the broker position as close as possible to the summarized virtual positions
    – each instance should assume that it’s orders go through as long as broker rules are not violated (can be checked to a major extend on the arrival of a virtual order at the order thread)
    – the order thread does not act immediately on virtual transaction requests, but collects them for a small time period before calculating the new resulting net position from virtual order tables and placing broker orders

    From my experience with a paper trading order manager I found it crucial for reliability always to construct a virtual order table from a transaction log (and not to populate a virtual order table directly).

    Best regards,
    Fd

    • admin says:

      Hi Fd,

      Thank you for your comments :o) I think I fail to understand several important things about your suggestion. For example:

      – how do you keep this transaction log in practice ?
      – how do you build a table from this log ?
      – how do you avoid the log becoming too large with time and table building therefore taking a lot of time/resources?
      – how do you manage to have a single thread that is able to properly enter multi-currency trades in MT4 if tick-by-tick synchronization is imposed on threads ? (threads that never end (calls to start() from start()) are also a bad idea due to stability reasons)
      – how do you avoid large requotes ?

      In my experience the use of one single thread for order synchronization is very problematic due to the fact that you need to enter trades on many different symbols. Polling of orders for some time may also be a bad idea because it may cause very large slippage in fast moving markets. However if you have been able to code something that deals with the above problems successfully I would be thrilled to hear about it :o) It would be great if you could send me the code if you have already coded something of this sort. Thanks again for commenting,

      Best Regards,

      Daniel

      • Fd says:

        Hi Daniel,

        please see my comments below.
        > – how do you keep this transaction log in practice ?
        memory copy and a csv file will do

        > – how do you build a table from this log ?
        a log entry would consist of all parameters of an OrderSend, OrderModify, or OrderClose. An order table consists of all entries available after doing an OrderSelect.

        > – how do you avoid the log becoming too large with time and table building therefore taking a lot of time/resources?
        You can throw away all entries from time to time that have no more relevance for the current virtual order table (archiving them could be a better idea to be able to track down possible errors). Volume of order logs can be neglected in times of GB hard drives.

        > how do you manage to have a single thread that is able to properly enter multi-currency trades in MT4 if tick-by-tick synchronization is imposed on threads ?
        I fail to see why multi-currency imposes an additional complication. If you do all communication over a proper IP library (e.g. ZMQ) you will have no problem with multiple threads communicating to 1.

        > – how do you avoid large requotes ?
        the frequency of requotes is dependent on the delay between receiving a quote and order request. Thus it is for sure dependent on the time window needed to establish synchronization (which of course can be tuned). As I’ve said, the order thread thread has to care about the requotes, thus it’s performance will not be worse than currently once the order is out for the first time (where the synchronization period comes into play). For practical reasons I think a window of 1-2 seconds should do. As our trades usually are running for hours or days, the impact of this delay can be neglected for sure. However, it is clear that an order manager can cost you money.

        > It would be great if you could send me the code if you have already coded something of this sort
        Sorry, I’m not allowed to.

        Best regards,
        Fd

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons