Last week I wrote a post about how difficult it can be to predict real out of sample profitability and why complicated methods such as walk forward analysis really do little to improve performance under unseen market conditions. However another way to attempt to predict out of sample profitability is to avoid looking at in-sample statistics and instead focus on general system characteristics, for example variables related to system complexity. On today’s post we are going to take a look at the Asirikuy price action trading system repository (containing more than 4500 trading systems at the moment) to see whether there is any relationship between the complexity of the trading strategies created and their out of sample profitability. This will give us interesting information about whether having simpler or more complicated strategies can affect out of sample performance in a predictable manner.
–
–
The general approach to live trading performance predictions is to attempt to relate in-sample statistical properties with out-of-sample success. For example you might want to take a look at whether systems with higher Sharpes tend to perform better under live trading conditions. You can use any in-sample statistics you want to attempt to perform this sort of analysis. However we can use non-performance based variables – such as system characteristics – to attempt to find these same relationships. For example we might want to see whether systems that trade at a given hour are bound to be more successful in the out of sample than systems that trade within a different hour. We might also take a loot at things such as the types of variables used to enter trades or the trailing stop used to manage them.
Perhaps one of the most interesting variables that we might think about is system complexity. Generally people expect simpler systems to do better and more complicated systems to do worse, in part because more complicated systems are more prone to data-mining bias and therefore more likely to be the result of simple chance and therefore to fail if no precautions against mining bias are taken. However since our repository creation process adequately controls for data-mining bias, regardless of system complexity, we can make some interesting comparisons to see whether there is any relationship between the complexity of the created strategies and their out of sample performance without worrying about data-mining bias. In the case of our price action based trading systems complexity is also really straight forward to determine since the systems use either 2, 3 or 4 price action rules to enter trades. The rule number is a very good proxy for system complexity.
–
–
For the above analysis I used systems with at least 5 months of real out of sample performance, which means that I only used systems that were created up to December 2015. At this time we had 1890 systems within the trading repository, with the large majority of systems created between July and December 2015. This means that the majority of systems have similar amounts of out of sample performance. Of course more complicated strategies are expected to trade less than less complicated strategies – because more rules implies higher filtering – so we must also take this into account when analysing the data at hand.
Since the most interesting thing we can look at is whether there is a relationship between losing and making a profit in the out of sample the first thing I looked at was the skewness of the OS profit distribution for systems with different levels of complexity (the first image in this post shows this comparison). It seems quite clear that more complicated strategies have a far more positive skewness than the less complicated strategies. There seems to be a higher tendency for strategies to reach a positive performance. However the kurtosis is also much lower since the distribution is much narrower, something that we can easily explain by accounting for the lower number of trades for higher complexity strategies. This can be easily seen in the second picture showing the distribution of OS profit for the three complexity groups (blue = 2 rules, green = 3 rules, red = 4 rules).
–
–
Of course it is difficult to know whether the 4 rule distribution will mutate into the 2 rule distribution as many more trades are accumulated, this might as well be the case and is a reason why we must interpret these results very carefully. Right now it would seem that rule complexity is directly proportional to a positive out of sample performance but it might just be that better out of sample results are expected at lower trade numbers (meaning that systems deteriorate as they trade, which is also something we expect). It is also worth mentioning that we can obtain net profitable outcomes even on only slightly positively skewed distributions since we never take the full losses of systems that have a negative OS (since they are discarded by worst case statistical criteria based on linearity and monte carlo simulations).
We can also look at the OS Sharpe to see if this relationship holds for a normalized criteria which is independent of the number of trades. The results – last graph in this article – also supports the idea that the more complex systems have a tendency to have better OS Sharpe ratios, significantly better than those of the 2 and 3 rule systems. However this still does not eliminate the influence of the absolute number of trades that have been taken, again the result could just mean that systems are expected to have better OS Sharpe distributions when they are taking their first X trades, rather than simply because they have N rules. It could be a story about complexity or it could be a story related to age which is simply related by proxy with system complexity, a mystery we’ll look into within a future post.
Of course if you would like to learn more about how our repository is constructed and how you too can use the power of GPU mining to construct and trade your own price action and machine learning based portfolios please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.
