The relationship between the last year of in-sample testing and the first year of out-of-sample in long term profitable systems

During the past month I have been puzzled by our findings on large portfolios suggesting that systems that have had positive results in the last year are more prone to under-perform during the following year (read here to learn more). This tendency can be attributed to mean reversion but it seems quite odd that the market would generally favor the choice of trading systems that were under-performing in the recent past. Today I want to share with you a more judicious analysis of whether this tendency is true or false, using pseudo out of sample testing in order to evaluate whether the above phenomena is actually true. As you will be able to see it is very easy to get fooled by randomness when attempting to discuss this sort of thing, reason why it is always important to perform larger scale testing and to avoid conclusions based on small amounts of data.


To evaluate whether highly stable (R²>0.95) long term historically profitable trading systems have a tendency to mean revert I decided to do an experiment using random in-sample period selections. For this I created uncorrelated and high R² systems (R²>0.95 and frequency higher than 10 trades/year) across randomly chosen 15 year in-sample periods, then looking only at the following 365 days after that period as a pseudo out-of-sample test. I generated 1295 trading systems in this manner – all with distinct in-sample periods – and then evaluated whether there is a relationship between the last year profit of the in-sample period and the results in the pseudo out-of-sample testing period range.

The results show that there is no strong correlation between the pseudo out of sample profit and the last year of in-sample profit when the in-sample period selection is randomized. The overall correlation between both values is 0.021 which denotes practically no relationship between both factors. This suggests that while under some years results may appear to be strongly mean-reverting, there might be others were results favor the most recently profitable strategies. Although results appear to be skewed towards the positive side, this is merely because the average last year profit is positive (mean +4.74%) and the majority of systems did have a positive out of sample result (mean +3.06%). This only shows that there is a global tendency to perform positively in the pseudo out of sample, something which is expected from trading strategies that have been developed with long term historical results in mind.


Looking at the correlation diagram showed above we can also see that the pseudo out-of-sample variable (highlighted by the black boxes) has a very poor correlation to all in-sample variables, including the last profitable year (darker colors mean stronger correlations). This is something I have observed repeatedly in the past as the prediction of short term results is heavily prevented by the market. If you were able to predict short term success by looking at some in sample characteristics trading would be very easy as you would be able to constantly select strategies that would be more likely to perform better in the short term. Although there is a clear tendency of systems to perform positively in the long term when they are developed across large amounts of market conditions there is nothing I have found that can predict whether they will perform profitably under the immediate conditions following their creation.

The above makes sense. A system that is adapted to a wide variety of market conditions might be able to profit in the long term because those conditions are very likely to show under long periods of time at one point or another but it is very difficult to predict whether a system will have short term success because the next set of conditions that the market will show cannot be reliably estimated. It’s like trying to guess which card will be drawn out of a deck of cards. You can guess with a high probability that the ace of spades will have come up after 40 draws but it is very difficult indeed to guess whether the first card drawn will be the ace of spades.


The above teaches us that it is always important to have enough data to be able to make some conclusions. While analysing small amounts of out-of-sample data might lead you to conclude that systems tend to mean-revert – as I showed in one of my previous articles – it is important not to imagine that those observations will be prevalent across all future market periods. While some short term out-of-sample periods might show some strong relationships under some market conditions, those relationships might be unique to that particular succession of market conditions and may hold no real truth for the overall market. Of course if you would like to learn more about how to perform analysis like this and how you too can create your own historically long term profitable systems please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.

Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

WordPress › Error

The site is experiencing technical difficulties.