When you build and design an automated trading strategy you certainly gain the ability to do parameter optimizations in order to find the set of variables that gives the best historical results. For example, if you build a strategy that trades a moving average cross you will be able to change the moving average types and periods and you will – most likely – be able to find a set of variables that gives better results than what you initially tried. The process of changing parameters to arrive at better results is called optimization – as mentioned above – and the main problem associated with this process is what people in the field call “curve fitting”. On today’s article I am going to give you an in-depth look into the understanding of curve-fitting through Fisher’s natural selection theorem, giving you a good idea of why it happens and the inherent assumptions made when trading systems which have been heavily optimized.

But what is curve fitting ? From a strict point of view, curve fitting is the adjustment of a trading system to a certain set of historical data in a way which causes the strategy to perform very well historically but fail under future market conditions. In essence what we’re saying is that a curve fitted system will perform extremely well under accurate historical simulations but will fail eventually when you trade them in the market live. It is very intriguing to understand why this happens and what can be done to prevent this. What causes a system to be “adjusted to fail” ? What is the inherent mystery behind the optimization of systems that causes curve fitting ? This is where I will go to Fisher’s natural selection theorem for some very interesting perspectives.

–

–

In evolutionary biology, Fisher formulated a very interested theorem (called Fisher’s fundamental theory of natural selection) which postulates that an organism which is very well adapted to a certain environment will have a high probability to fail under changing environments. This is actually a very profound statement because it allows us to predict the ability to survive of any organism based on its level of adaptation to a particular environment. To illustrate this theorem let us think about the giraffe. This animal has evolved extensively to adapt to the conditions in the African savannah but this has made it too dependent on its surrounding environment. If all tall trees died and only small shrubs would remain the giraffes would probably die as they are so well adapted to a certain environment that they are unable to adapt to even small changes.

Fisher’s theorem is what I view as the corner-stone of the “dead-end” problem in evolutionary biology. Evolution works to take organisms outside of bad conditions but it necessarily doesn’t put them in positive evolutionary paths, something which is dependent almost entirely on chance. When an organism starts to evolve it does so because evolutionary changes have been favoured in some way but these changes can cause dramatic effects in an organism – which increase its medium term chances of survival – but make it potentially and dangerously susceptible to changes in environmental conditions. Again, the giraffe shows us this case perfectly, an animal which has evolved so radically that it has made itself significantly vulnerable to environmental changes.

But how does this relate to trading? Well curve-fitting when doing optimization is very analogous to “adapting a system” to a certain set of historical conditions. When a system gives extremely good backtesting results but it fails bluntly in live trading it doesn’t mean that there was something inherently “wrong” with the strategy but it means that the system became a “giraffe” in the world of trading systems, to “stiff” to adapt to changes in market conditions. Therefore Fisher’s theorem seems to apply perfectly to the world of trading as well, system which are very well adapted to certain sets of market conditions have an inherently higher probability to fail when the markets change. The smaller the change that is needed for your system to fail, the higher its chance of being curve fitted.

Curve fitting – as we had discussed before – goes hand in hand with system robustness. Take for example a simple moving average cross that works on 8 different symbols with the same parameters and a system that trades a very specific CCI cross with a fixed SL/TP and works only on one single pair during the past 10 years. In order for the first strategy to fail the behaviour of 8 pairs would need to change to a whole new behavior set (because if they interchange the system would still be profitable) while for the second system all that is required is for the market to change such that turns happen at a different CCI level on one single pair. The pair could simply start to behave like another one and the strategy would go down the drain. In essence the changes in market conditions for the first system to fail are dramatic while for the second one they are small, the second system is therefore more curve fitted and the first one is less so.

The most intelligent approach seems to be to think about trading systems as evolutionary systems with more adapted systems having higher changes of better short/medium term profits but also higher chances of long term failure while less adapted systems have inherently lower profitability with a much lower chance to fail going forward. Therefore depending on the choices you make you might have different results, choose a giraffe and you may be able to reap high rewards while the tall trees are growing while choose the sea turtle and you’ll be able to move slowly through the markets for a probably very long amount of time. Which way is better ? Well it depends – just as much as in biology – on how long a particular set of market conditions last. Using a highly adapted system might yield much bigger rewards if the conditions remain while trading a less adapted system might yield much lower gains. On the other hand if conditions change the high yielding system might quickly go to its formal statistical worst case scenario while the less adapted system will keep on producing profits for a much longer amount of time.

–

–

Some people might then argue that the best thing is to use a highly adapted system that is periodically re-adapted but this is still a highly adapted system since the level of adaptation remains and you’re always inherently lagging the market conditions you’re adapting to. If you re-optimize a strategy in order to “adapt” to current market conditions in reality you’re not diminishing your chances of failure as you’re simply trading within a part of your optimization space which is just as small. The problem is not the particular parameter set you’re using but the inherent level of adaptation within the strategy’s development process. A highly adapted strategy will always be “highly adapted” to a given environment, no matter how many times you change the environment you want it to be adapted to. Changes will always bring a higher chance of failure. As in evolutionary biology the best thing we can do is evolve systems out of bad historical results under past market conditions but we cannot be sure we’re putting them in a path which is right for the future.

An important thing to remember here is that both paths can have their rewards and since the future is unknown there is no way of saying which might be the best way to go. However it is absolutely critical to remember the assumptions made when using highly adapted systems and to be aware of the fact that failure deriving from changes in the system’s ideal “space” will have higher or smaller changes of happening depending on how well adapted the strategy actually is. It is also important to consider that you cannot put all strategies in two distinct and separate categories but this is most likely a slow gradient spectrum that changes slightly with things such as backtesting period length, degrees of freedom in optimization, optimization steps, out of sample period length, etc.

If you would like to learn more about algorithmic trading and how you too can design and trade automated systems based on knowledge and understanding please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)

Now for the golden question, how do you empirically calculate everything you just said? Without some sort of numerical comparison we are back to guessing.

There is of course a way but we will need a bunch of super computers :) Calculate every combination of parameters (which will be a few million to billion). Then choose the strategy which lies in the middle (should give you the best of both worlds).

We will need some sort of practical way to calculate a “cruve-fit” parameter as the above method might be impossible if you don’t work for NASA.

Hi Franco,

Thank you for your post :o) Actually calculating the whole parameter space shouldn’t be a problem for many strategies with low number of variables. If you can have backtests for the whole parameter space you could determine a proxy for robustness by a simple equation like this :

RobustnessProxy = pp*y*s/(ps)

pp = number of profitable sets

y = years

s = symbols

ps = number of total sets (parameter space)

If you run this equation assuming the minimum possible parameter variations and the largest number of available symbols you can get a global comparison for the robustness of any strategy you come up with. Of course, you would be limited by the computational effort required to run these backtests (as you have mentioned). The above equation is also “nice” in the sense that it doesn’t punish degrees of freedom as long as they are profitable. Thanks again for posting!

Best Regards,

Daniel

Interesting concept thanks for sharing! That formula might work for overall robustness, but how do we determine which parameter set will give you optimal results (balance between robustness and profitability). A robust strategy that makes 1% per year for the next century is useless, as well as a strategy that makes 100% per year and then fails. We need to find a mathematical way to determine the parameter set which theoretically optimizes both criteria.

Might be somewhat impossible but that is the only way we will stop guessing. Current optimization methods that we use might increase the probability to lower curve fitted parameter sets but at the end of the day there is still a large guess involved :)

Hi Franco,

Thank you for posting :o) The fact is that the only way to stop “guessing” is to know the future and since we don’t we must simply do our best to manage risk in what is the optimum. You could certainly get measures of robustness and therefore know which systems have a higher probability to fail but knowing the exact probability is out of the realm of possibility since the future is basically unknown. I would love to “not guess” but such is the nature of this business :o) I wouldn’t call it “guessing” but more like attempting to get a result for a problem in which you don’t know all variables. It is more analogous to predicting the weather (finding meaningful patterns in a partly chaotic system) than to attempting to know what number will win the lottery based on historical results (which is guessing at best since this is -in theory- a chaotic system). So in the end we need to focus on managing risk and taking decisions based on the assumptions inherent to them. Thanks again for posting!

Best Regards,

Daniel

Thanks for the reply :) I’m not interested in exact probabilities as we all know that is impossible, but some sort of empirical indication would be helpful. At the end of the day we should do the most in our power to beat the odds. I’ll think about this issue maybe we should start a thread at the forum to bundle up some ideas as this is obviously a very important topic