Mechanical Forex

The choices we make when develop trading systems from historical data can lead to some significant problems. The way in which you develop your strategy creates several types of statistical bias that affect the quality and eventually the probability that your systems will work when tested under real out-of-sample conditions (new live market data). Understanding the different types of bias that are generated and how these types of bias can be controlled, reduces or eliminated is very important for the development of reliable trading strategies. On today’s post I am going to talk about the different types of statistical bias in trading system creation and how you can control these types of bias by modifying your trading system development process. Although this is no guarantee that your process will lead to profitable systems it does eliminate a big factor in the potential failure of such strategies.

–

–

The term statistical bias refers to any factor that makes the expected value of what we’re studying deviate from its real value. In trading system development it is anything that makes us believe that a system is bound to perform in a certain manner when in reality it will perform very differently. There are some types of bias that we can control more effectively than others – some we can come close to eliminate – while there are others which we cannot effectively tackle as we run into the problem of simply not knowing what the future holds. I will now go into the two of types of statistical bias that I consider most relevant to trading system creation and how they can be attacked. I will try not to give them specific names since different people often use different names interchangeably for very different types of bias which might make naming problematic. Therefore I am going to refer to these bias types as type I and type II.

The type I bias is the statistical bias created by the intensive search for a system with a given set of performance statistics using the same historical data. When searching for a strategy that fits a desired level of performance, the continuous and intensive use of the data plus the degrees of freedom added to the system generates a real probability that a historically profitable strategy with the desired characteristics is found spuriously. This means that the strategy you found does not correspond to some true ability to predict the future using historical data (a real inefficiency) but simply a system that generates a given result due to randomness (you tortured the data enough, you got an answer). The type I bias therefore deals with the probability that your searching method will generate a system with the sought level of performance just by random chance and therefore it can be controlled by looking at the probability to generate a system with the same performance on sets of data where no causal relationship between the past and the future exist.

To evaluate for the type I bias you can generate random data series using boostrapping with replacement from the return/range distributions of the original data series, which generates a time series that has all the same characteristics (distribution wise) but has no causal relationship between the past and the future because this relationship has been destroyed by the shuffling of the returns/ranges. Some time-dependent features of financial data, such as the autocorrelation of bar ranges, are obliterated within a random data set created in this manner. You can then perform the entire system building process on the random data to attempt to find something that has the same performance as your system created on the real data. This is clearly very difficult in a manual system creation process – as repeating the same system creation process on random data is often impractical or simply not possible – but it is absolutely possible when the logic is generated algorithmically either by data-mining of specific rules or by using some more elaborate data-mining techniques such as machine learning algorithms. Type I is often unknown when systems are created manually.

After doing the above process on a large number of random series you can therefore calculate the probability that the performance obtained on the real series comes from random chance. If the probability is below what you consider an acceptable threshold (I generally 1%) you can consider that this bias is low and that your strategy actually found a real historical inefficiency within the data being used. However this says nothing about the probability that your strategy may perform in the same manner going forward, it merely tells you that you found something that is statistically relevant within the historical data.

–

–

The type II bias deals with the problem of your system not finding something general but something specific that has a high probability of not working going forward. This bias can be better understood by a simple example. Imagine you create a system using one year of historical market data. You have created the system through a mining process and by doing tests on random series you have found that the probability of your system coming out of randomness is very small (let us say less than 0.0001%). However once you start trading your system live, it simply fails, so you wonder what actually went wrong. What happened in this case was not that what you found within the historical data wasn’t “real” but that it wasn’t a general characteristic of the data but something that was limited to the year that you studied. You found something very specific that wasn’t general to the whole data.

This new bias type therefore deals with the finding of something that is general to the behavior of the financial instrument being studied and not something specific to the market period that has been studied. It may be clear that this bias is ever-present and cannot be eliminated since the number of future market conditions is always above the number of market conditions we have for study (our historical data). However we can attempt to attenuate this bias by developing systems that successfully tackle all historical market conditions we have at our disposal. By doing this we ensure that we capture something that has been relevant at least for all the data we know. This is of course no guarantee that the system will be profitable – as we can still face something outside of the historical data – but at least we know we are covered for all the market has experienced within our available history. It is obvious that something that has worked on a historically real inefficiency that has been present over 30 years of data is more general that something that was found using only 6 months of data.

As you can see the difference between these two bias types is important and yet subtle. Type I deals with whether something we find is actually statistically relevant within the real data and not something we found simply due to random chance while the second type of bias deals with the probability that the system captures a general aspect of the underlying series instead of just a specific aspect that is not a general characteristic of the financial instrument being traded. Control both bias types and you have a good chance that you have found a historical inefficiency that is both real and likely to be a general characteristic of the financial instrument under study. Will it be profitable? No one knows the future :o).

If you would like to learn more about how to build trading systems with controlled statistical bias and powerful data-mining techniques please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)