Last year – if I remember correctly – I wrote a post about the possible usage of the chi-squared test as a way to determine whether or not a system was following its intended statistical distribution of returns. However after some testing it became evident that the chi-squared test is not easily applied to any trading strategy as the test has some inherent limitations which make its application meaningless within certain strategies. On today’s post I want to talk more about how to properly use the chi-squared test as a worst case scenario criterion in Forex trading and why it is a very good compliment to other criteria such as worst case draw down values derived from Monte Carlo simulations.

First of all we should remember what the chi-squared test is all about. When we have a given distribution that must be fit by a set of data but we do not have enough data points as to “fill” the entire distribution in the way which it should get filled in the long term we are faced with the problem of knowing whether or not our limited data set fits or doesn’t fit the intended distribution, even though the current amount of data is less than that which is needed to show the true character of the distribution. For example if you have a distribution which shows that in the long term 20% of trades are between -1 and 0%, 40% of trades between 0 and 1% and 40% between 1 and 5% but you only have 5 trades which have a -0.5% outcome it becomes critical to know if this distribution still fits a particular case of what would later become the target distribution. The chi-squared test does precisely this, allowing us to know how well a test distribution fits a target one within a given confidence interval.

–

–

In the ideal case things are not very complicated, we have a given set of trading results derived from backtesting which form a long term distribution that our live testing should match and if the chi-squared test at a 99% confidence fails we know that our current results do not match the distribution we want. It is a distribution based criteria which tells us to “stop” the trading of our system since it isn’t what we have been statistically preparing to trade. The problem that we face is that generally this is not the case as the chi-squared distribution does not work well with many types of systems. The test fails whenever there is a very low number of classes (for example when a system has a fixed SL/TP) or when there are some very odd classes which are fundamental to the description of the system (for example if you have a system like Quimichi or Ayotl where very rare but extremely profitable and variable trades are a fundamental component of the trading strategy).

How do we solve this problem ? Well the solution comes in the form of Monte Carlo simulations. In order to know how to apply the chi squared test we must carry out Monte Carlo simulations based on the historical back-testing distribution using the number of trades we have in our live data (the one which we want to test) and after applying a chi-squared test to all outcomes of the Monte Carlo simulation we will be able to know if the chi-squared test is predictive or not of the success of the actual fit of the number of trades we currently have against the long term distribution. For example if I have made 100 live trades of a portfolio and I want to carry out this test I run a 100K iteration distribution-based Monte Carlo simulation and I calculate the percentage of iterations that passed the chi-squared test. If after doing this I see that less than 99% passed the test then the chi-squared test is NOT a good predictor of system success with this number of trades.

This powerful Monte Carlo technique allows us to only use the chi-squared test when it is a relevant predictor of actual distribution quality. If I run the Monte Carlo simulation and 100% of iterations with 100 trades passed the chi-squared test but my 100 live trade fails it then I have a very strong reason to believe that my live trading is very far away from what I would consider normal behavior for my trading strategy. As you can see the chi-squared test on its own is not very useful but once it is applied through Monte Carlo simulations we can definitely know if it does predict distribution matching, giving us a formidable new weapon to stop the trading of our systems if they fall outside of what the simulations tell us.

Our Asirikuy Monte Carlo simulator already has the ability to calculate the percentage of successful chi-squared tests across all iterations of a simulation so in Asirikuy getting this results is as easy as loading the backtests on the draw down tool, exporting the distribution classes to the Monte Carlo simulator (which is done automatically) and changing the number of trades to match the number of live trades we want to test against. If the chi-squared results show high degrees of success we can then calculate the distribution of returns from our live results and instantly know whether or not our system has gone outside of what would be considered normal for it. You’ll see that for many trading systems and portfolios a significant number of trades is needed in order for the chi-squared test to be of any importance as enough trades have to be made to have a good enough initial “fill” of the distribution space.

If you would like to learn more about Monte Carlo simulations in trading systems and how you too can learn to derive important worst case information from this formal statistical tool please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Brilliant idea Daniel! Will we be able to calculate the distribution of live trading results in the new tool?

I was wondering the same as Franco, that I wouldn’t know how to calculate the distribution of live trading results, let alone how to load them into the profit and DD tool…….

If this was easily done, then as you say it would be another very powerful method of knowing when to stop running an account……..

Calculation is easy it is that you don’t want to do it by hand every time!

Well, this is another brilliant idea, thanks for the explanation, but I will have to read a few more times to digest it.