Back/Live Testing Consistency: Using Different Brokers For Long Term Simulations and Live Trading

Through the past few weeks a group of Asirikuy members manifested a lot of interest in the issue of back/live testing consistency and because of all their questions I decided to release a training video in Asirikuy dealing with the comparison of live and back testing results. The video dealt with how these comparisons should be carried out and what a user should look for to know that there is some consistency between simulations and live performance. After this video some interesting discussions started within the Asirikuy forum dealing with what happens when you trade using a broker which does not match the long term data you use for live testing and how this can contribute significantly to variability in live trading statistics.

Certainly when you create an algorithmic system based on 10 years of back-testing data and obtain certain profit and draw down statistics you would expect to use these statistics as a clear representation of what you might expect in live trading. Unfortunately the sources of back-testing data are limited, and – in Asirikuy at least – we only have access to 11.5 years of Alpari UK 1 minute data meaning that every time we intend to run a system on a set of data different than Alpari UK we will probably have some deviations from back testing results, mainly due to the fact that brokers have several characteristics which make them different (feeds, week opening times, number of candles per week, 4H candle structure, etc).

However it becomes important to know how different results might be and how consistent Alpari UK live tests to back-tests actually are to know how much our statistics deviate and therefore how bad the approximation of developing a system using a broker’s long term data and then trading systems on another actually are. It is also worth mentioning that Asirikuy systems are well behaved strategies which operate only above the 1H time frame which use very high profit and draw down targets which completely eliminates the possibility of inaccurate simulations (the system are developed to give accurate simulated results). What we are going to evaluate here is how well a live test matches a back-tests of the same period, noting what the difference might be if the live period belongs to the same backtesting data (Alpari UK) or to another broker.

The image above shows the results of an Asirikuy trading system (almost 1 year live test) against the results of a backtest of the same period with the exact same account characteristics and EA parameters. You can see that when the data from the same broker is used we get results that clearly match between back and live testing which implies that the 10 year results have a high degree of validity. Certainly running systems on Alpari UK developed using Alpari UK data is a very good way to preserve certainty regarding the system’s statistical characteristics. As well as the systems are well-behaved (developed on upper time frames with no interpolation problems or other exploitations of backtesting limitations) we will have a good estimation of statistics through our backtests.

However when we compare a backtest using our Alpari UK data with another broker (in this case FXClearing) you can see below how the system (the same as above) has results which are very deviated from the backtests. In this case the deviation is negative but this does NOT mean that this will always be the case as there are some instances when the case is completely opposite (the other broker’s live results have much higher results than the backtest). The important thing however is not the positive or negative variation but the fact that there is a higher degree of uncertainty regarding statistical characteristics, meaning that for certain systems running a ten year backtest on one broker’s data and then running it live on another broker will imply somewhat different results. Note however that results are still profitable (meaning that the inefficiency is still present regardless of broker dependency) but profit to draw down ratios are affected in the short term (we still do not have enough data to know whether or not there might be long term convergence to the same profit to draw down ratio).

It is also important to consider that this does not happen for all systems as other Asirikuy systems have very well correlated results with Alpari UK data. This is mainly due to the fact that these systems are less sensitive to broker dependency issues which arise from broker differences which can be as simple as the starting hour for the week. However the problem here is clearly that we can have important degrees of uncertainty regarding the precise statistics of trading systems if the running broker and the broker from which  we got the LONG TERM back-testing data do not match. In order to fix these problems we will seek to implement mechanisms within the F4 framework (the next step in Asirikuy EA development) which will make sure time frames are rebuilt so that they ALWAYS match the overall bar number, weekly starting/ending times and other such characteristics of the back-testing data.

Right now however we know that  Asirikuy systems (which are developed with very sound tactics in mind) preserve their profitable character even across this type of dependency in statistics so we know at least that the steps we have taken to ensure reliability can preserve the mathematical expectancy of strategies across different brokers. However right now we’re very aware of these issues and the next versions of Asirikuy systems (F4) will have very powerful features to equalize all broker charts to drastically reduce dependency. It is however important to consider that for systems that are developed on lower time frames which depend on much more variable broker characteristic a change between the long term backtesting data source and the live executing broker can imply a 180 degree change in profitability (which is also greatly affected by execution issues that hardly affect Asirikuy systems). This is yet another reason why it is better to use higher time frames for algorithmic system development.

If you would like to learn more about my journey in automated trading and how you too can develop systems based on sound trading tactics with reliable simulations in mind  please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons