On Yesterday’s post I talked about the Ulcer Index and how this measurement – along with the Pain Index – allows us to see how easy or difficult it is to trade a strategy from a psychological point of view. Both of these indexes use draw down period lengths and depths as a way to tell the trade whether a strategy will be very “stressful” to trade on an account. Thinking about this measurements and how their pertain to a non-profit related aspect of system quality I thought that we have indexes to measure almost every aspect of a trading system except the robustness of a trading strategy. Through the following few paragraphs I will introduce you to an idea I had – The Hardness Index – which allows us to measure how robust a strategy might be when compared to another one.
When we think about how good or bad a trading strategy is we usually think in terms of draw downs and profits. The only thing that most people – especially new traders – tend to value is the ability of a trading system to bring profits with limited draw downs. The problem with this is that many – if not every time – most of the statistical information used to derive these values is a consequence of simulations and therefore constitutes a measurement of expected profit Vs real risk and as a consequence we may fool ourselves to believe that a system is “very good” when in reality it is simply not.
A very important characteristic of a strategy – beyond how much profit or draw down it brings – is its robustness which can be defined as the like hood of a strategy to continue to work under future market conditions. Certainly there is no “crystal ball” to tell us if a system will fail or continue to work but there are several things we know which we can use to mathematically measure the like hood of a system failing against another. For example we intuitively know that a strategy which has reliable profitable simulations on 10 different instruments is bound to be less prone to failure than a system which trades a single currency pair and we also know that a strategy which has 20 variables optimized using a 0.01% step criteria has a high probability to be curve fitted (and therefore to fail under new conditions).
Using this information I have devised an equation – the Hardness Index – which tells us how a strategy compares to another when talking about their robustness. The equation takes into account all aspects I consider relevant to system robustness, in particular when such strategies are derived from simulations. Note that this index does NOT contain any profit or draw down information but measures trading system quality simply through the like hood of the strategy being able to withstand future conditions. This is how the index is calculated :
The higher the value of the hardness index, the more robust a strategy is and the higher its quality is regarding the possibilities it has to survive future changes in market conditions. As you see a system that trades more instruments with the same parameters, has more out of sample years and more coarse optimizations has a much better chance at surviving in the future. The index puts a lot of weight in out-of-sample testing as this shows in fact if the strategy survived or not to changes under market conditions it was not optimized for. The hardness index rewards systems which have less degrees of freedom and longer testing periods while it “punishes” strategies which have a higher like hood of being curve fitted (finer optimizations, shorter testing periods, more degrees of freedom, etc).
The value of the index can vary significantly between systems. For example let us consider the current Quimichi strategy – a very robust system based on 7 different currency pairs – and calculate the pain index for it. The solved equation is showed below :
The result above shows that the hardness index for Quimichi – taking into account its development process – is 466 while the hardness index for a system like Watukushay FE which trades only on the EUR/USD and had many more variables optimized in a much finer manner has a hardness index of merely 8. This definitely shows that the difference between the robustness of both systems is very large and Quimichi may easily be considered far more robust as it trades more currency pairs, had less variables optimized and those variables were optimized in a much more coarse fashion than for Watukushay FE.
Although there may be several ways to improve the above proposed calculation, this Hardness Index will be able to give us a good idea of where different strategies stand in relation to how robust they might be towards changes in future market conditions. The above index is my first attempt to ever measure robustness in a mathematical way, something which is necessary since robustness is a very important characteristic which is often neglected in favor of other statistics such as profit to draw down ratios while it might be one of the most important characteristics of a trading strategy. Of course this is just a proposal and any suggestions or contributions to the above equation would certainly be welcome so feel free to leave any comment, question or suggestion you may have !
If you would like to learn more about my work in automated trading and how you too can design trading strategies with robustness in mind please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)