Yesterday I talked about the amount of time that it takes linear correlations between in-sample and out-of-sample performance to show up when live trading a set of price action based strategies. It was evident from this analysis that it would take years before a clear simple relationship could be drawn and therefore attempting to find such simple relationships before a significant amount of time had passed seemed like an exercise in futility. However we might be able to make simpler predictions using more complex non-linear models to classify the out-of-sample performance of live trading strategies. Today I will talk about the result of a classification model using real out-of-sample data gathered through the past 6 months from our price action based live trading repository.
–
–
In the past I have always failed to come up with linear or non-linear models that have any edge in attempting to predict strategy out-of-sample returns. However this week I decided to try a new approach – non-linear classification – to see if I could at least predict if the average return of a strategy was expected to be negative after more than 6 months of trading. To do this I used a random forest classifier coupled with both in-sample statistical information and information relevant to the characteristics of the trading strategy used (stop magnitude, trailing stop characteristics, trading hour, etc). In the end I had 25 different characteristics I could use to see if failure to profit in the out-of-sample could be predicted.
The number of trading systems I used for this analysis was 1000 – the amount that fulfilled having more than 6 months of live trading – which gave the random forest engine enough information to generate the algorithm. This set was randomly split into two sets, a training set with 80% of the systems and a testing set with the 20% remaining strategies. The prediction objective was simple, predict whether the strategy would have an average positive or negative daily out-of-sample return. The random forest algorithm was generated over the training set and then its accuracy was evaluated over the testing set.
–
–
The results in the first table displayed above show you a typical result from generating a random forest model to make this prediction in the testing set. As you can see there is an increase in the number of winning systems within the testing set when using the random forest prediction Vs trading all systems and – most importantly – the actual out-of-sample average daily return increases by 29% over the average daily return when not using any prediction algorithm. This means that when using the prediction not only were you able to discard more unprofitable than profitable models, you were able to discard the models that would have had the most unprofitable results.
Furthermore expanding the test environment to the entire set of available systems – that contained an additional 1700 strategies – still revealed an increase in the number of winning strategies by 2% and an increase in the average return of almost 10% (relative to the return obtained by trading all available systems). This means that even though these strategies have yet to reach the 6 month mark where the predictions are expected to take the most effect the model is already able to discard some strategies successfully from the set. Of course the model has much more trouble when faced with far shorter term returns – the most common casein the full set – since the effect of randomness on these returns are much greater due to the magnitude of the traded edges (as I explained in my last post).
–
–
The importance plot showed above is also very interesting as it allows us to see the relative importance of the different variables within the random forest classifier, a great advantage from using random forest models. The in-sample profit factor, CAGR and maximum drawdown are the most important factors regardless of the fact that the linear correlations of these variables to the out-of-sample average return is almost 0. There indeed seem to be some non-linear relationships that can be used for the successful (at least to some extent) classification of systems into profitable/unprofitable sets, regardless of linear correlations being non-existent. Sadly these relationships are not strong enough for the construction of non-linear regression approaches – as I have showed previously – but they can work in classification as showed herein.
It is also worth noting that the random forest model is non-deterministic, meaning that every time you generate a model it will create another different random forest. Despite this fact after 200 model generation attempts – with different Train/Test random shuffling sets – the results were always very similar, showing that the results in this article are not from a “statistical fluke” inherent to the model generation process but stem from true relationships within the data. Granted the above could be a model that only applies to the period where the strategies have been currently live trading (first semester 2016) so we still don’t know if it will hold up in the future – if it is indeed a general model for OS return classification – but it is very encouraging to finally see that at least there are some machine learning models that can come up with successful predictions. We will track this type of model going forward to see whether it does actually provide some edge. If you would like to learn more about trading system creation and how you too can create and manage trading system portfolios please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading strategies.
