Choosing Portfolios Without Bias: Some Strategies Against the “Cherry Picking” of Systems

When building trading system portfolios we encounter the interesting challenge of choosing a combination of systems to give us the best possible results under future trading conditions. It is sometimes easy to misinterpret this challenge as a challenge to get the “best possible back testing results”, something which causes the generation of cherry-picked portfolios were systems are picked due to their great “matching capabilities”. Although this method tends to give the best historical testing results it might generate problems under future market conditions since the system selection could have been “curve fitted” to achieve the best possible portfolio results in the past. It is therefore important to find other selection criteria which might allow us to create portfolios without the possibility of any selection bias.

The first thing I want to be clear about is that there is still no evidence which tells us that the cherry-picking approach doesn’t work. When selecting systems to build a portfolio your selection of possible strategies is limited and therefore when you pick systems that compliment each other to create a portfolio you’re implicitly introducing a hedging element which will protect your equity under many different sets of market conditions. If the cherry-picking process is done carefully you can avoid excessive matching of profit/draw down periods and create portfolios which are likely to be robust and survive future market conditions (because the failure of a system is bound to cause great success on another).

The problem with the above approach is that it doesn’t allow for a standard criteria for portfolio building and therefore there is no “set way” in which you can build a trading setup which ensures that robustness will be achieved with a high probability. Cherry-picking’s success depends on the experience of the portfolio builder and the overall understanding he or she has about the underlying characteristics of the systems being traded. Some traders who are not very experienced may inadvertently build a portfolio with great looking historical results with some very unsound historical correlations that may easily manifest themselves in the future. I therefore view cherry-picking as a sort of discretionary portfolio building approach which does not depend on any single criteria but on the analysis and good judgement of the person doing it.

Due to these reasons I have decided to build a portfolio approach which does not depend on good judgement but on some predefined criteria which does not allow for curve fitting strategies. In essence the simplest ways in which this can be done arise from the system building process. When building systems to allow for portfolio development without cherry picking some standard procedure for the system development effort must be established and then a simple criteria to choose amongst the results must be generated. In the case of Coatl – where a standard system development procedure exists – we simply selected baskets of strategies centered around single currencies which allowed us to generate trading setups without cherry picking.

The standard way in which portfolios are developed to allow for the greatest reliability involves the creation of a strategy that trades on many different currency pairs, then choosing pairs according to some criteria. There are several ways in which this can be done involving either the spread, liquidity, single centered, etc. In essence what you do is to develop a system for a given set of pairs and then select from those without cherry-picking (saying for example, I will trade on pairs which have spread below 5 pips). There will be a great incentive to only build a portfolio with the most profitable outcomes as the lack of a cherry-picking bias will often lead to portfolio selections with less than optimal backtesting performance (although with a higher inherent robustness) but this has to be avoided in order to rip the benefits of none discretionary portfolio development.

In my eyes probably the most iron-clad way to develop a trading setup would be to simply come up with a system and trade it with the exact same parameters on all pairs. This has the inherent highest robustness (probably there is only an extremely small chance of the setup not working in the future) but the portfolio will be suboptimal and greatly under fitted. It is important therefore to recognize that selection in portfolio creation is important to achieve the best balance between doing no selection and cherry-picking. You need to understand that a selection criteria should exist but the best thing is for that criteria to be unrelated with performance variables.

Another important thing is to avoid fitting the selection criteria against performance variables as this would totally defeat the purpose. For example if you have developed a system which trades on 10 instruments and a spread related selection criteria yielded suboptimal results but some sort of made up criteria related to swaps yielded the best historical performance results you could be simply trying to find some sort of criteria to satisfy your implicit cherry-picking. In the end every sort of selection criteria introduces bias – because you are selecting a selection criteria – but this effect should be minimized by an acknowledgement of all possible selections, meaning that you should consider not trading a given setup if every other setup gives worse results (because you would be cherry picking if you chose such a criteria).

Certainly the above methods of selection are only valid when you have a standard method of system development or when you have a single system that “works” on several different currency pairs with cherry-picking being the only alternative when you’re dealing with single-pair strategies which can be freely combined with others. In the end cherry-picking – when done correctly – can lead to robust portfolios but using selection criteria which are not tied to performance evaluation can – when its possible – provide an added degree of robustness.

If you would like to learn more about my work in automated trading and how you too can get a true education in portfolio building and algorithmic strategy development please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)


Print Friendly
You can leave a response, or trackback from your own site.

2 Responses to “Choosing Portfolios Without Bias: Some Strategies Against the “Cherry Picking” of Systems”

  1. Hi Daniel-

    Good article and I like where you are going with this.

    We routinely “cherry pick” stocks, cars, jobs, real-estate, food and women – if only it were that easy! Nearly every choice we make has some qualitative criteria versus strictly quantitative. But the scientist in you is not satisfied with that solution – and that’s what we pay you for!

    In the end I think the past is the best and most significant indicator of the future. I suppose statistics plays an important part as well, but that’s not my expertise. So here’s the rough formula I have in my head for picking among the various systems on Asirikuy:

    PortfolioSelectionMetric =

    [Instances] + (Pairs/Systems in a portfolio – the more the better)
    [Compound Annual Growth Rate / Maximum Drawdown Ratio] +
    [Total Live Trading Return while on Asirkuy] +
    [Last 3 Months Total Return]

    Using a formula like that, the best systems on Asirikuy would naturally rise to the top.

    Past performance is no guarantee of future results of course – but it’s the best indicator that I know about!

    Let me know what any comments on that,


    • admin says:

      Hi Chris,

      Thank you for your comment :o) I understand why you have chosen these criteria to select Asirikuy systems when building portfolios as it makes sense from a historical performance perspective. I have to say that I tend to balance historical performance with other type of data which may not specifically pertain to profitability. For example using live results – especially the last 3 months – is something I would rather not do as you run the risk of cherry picking systems which have a high probability to enter draw down periods as we know – from statistical analysis – that systems which have had strong periods of profit tend to have a higher chance of going into draw down (also the steeper and longer the profit the more “overshot” the system is and the more likely a “correction” is due).

      Another thing is assuming that more systems on portfolios is a “better thing” (the more the merrier?), on tomorrow’s post I will talk a little bit about the problems of “big portfolios” but I would rather have a portfolio with limited number of instances selected based on their matching (based on measurements of statistical correlation of draw down and profitable periods). In my experience it is better to use fewer systems that “compliment” each other better than many systems just for the sake of using as many as possible.

      It is also important to consider that diversification in pairs may not necessarily imply protection from correlated draw downs. Some systems can trade a variety of pairs with extremely high draw down correlations meaning that the benefit from portfolio coupling is minimal since your draw downs will tend to align. It might even be better to trade several uncorrelated system son the same pair rather than a pair diversified portfolio with correlated systems, obviously the idea is to have both (many pairs, low correlation). You should also consider the magnitude of the edge a system needs to succeed, systems that trade more need to have a larger edge so portfolios of less frequent traders should be generally considered more robust (pairs with lower spreads should also be favored).

      To sum it up I believe that a consideration of the “best up until now” performance results are bound to generate some portfolios with great historical behavior with a great chance of backfiring on you due to the inherent cherry picking of systems with higher probabilities to face immediate draw downs. If you pick the best recent live performers you risk a higher chance of getting hit by a highly correlated draw down. As always it becomes evident that cherry-picking is not a simple process and adequate consideration of many variables (particularly those related with robustness) need to be taken into account if you want to ensure a higher probability of survival. Of course this doesn’t mean you need to choose recently losing systems but you need to factor in all the above in order to pick strategies that will yield the best possible hedging of profit/draw down periods. I hope this has answered your inquiry :o)

      Thank you very much again for your comments :o)

      Best Regards,


Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons