A little bit more than a week ago (here) I started a series of posts about the automatic creation of mechanical trading strategies and how this can in fact be done by a computer. There are many ways in which we can tell a computer to design trading systems but none of them is as powerful or efficient as genetic programming. In order to understand all the future posts I will write on this subject it is important to understand the concepts of genetic algorithms and genetic programming and how they can be used to create automated trading systems without – almost – any human intervention. More importantly it is vital to explain some of the problems of this approach and how they can be eliminated to obtain very robust trading strategies that have a high like hood of succeeding in the future. Today we will learn a little bit about genetic programming, what it is, some of its limitations and what it can do.

What is genetic programming ? In order to answer this question we first need to think about genetic algorithms and their definition. Several years ago programmers started to look for more efficient optimization methods for problems where very large amounts of variables – and variable combinations – needed to be tested against a particular set of data. One of the most successful approaches to find a very good solution to the problem took a hint from evolutionary biology. In nature the “best fit solutions” are selected through environmental stress, giving rise to new adaptations and species merely through a selective process.

When dealing with large optimizations we can take a similar approach (known as genetic algorithms). If 5 variables need to be optimized against each other and each variable can have 10 possible values genetic algorithms tell us to randomly select X combinations and then – from those combinations – select and “breed” the most successful ones to create a new group of potentially more successful combinations. From those you select the best ones, combine, then select the best ones, combine, etc. Each step in this process or new set of parameters is called a “generation” which was spawned from the previous most successful combinations. Sometimes “mutations” – which are just small perturbations in the set- are introduced to new generations to better imitate the natural selection process. The image below illustrates how this would work for the above mentioned example.

–

Computer scientists found after a while that the genetic algorithm methodology allowed them to find solutions to very complex problems after only a given number of generations. This approach saves an enormous amount of computer time when compared to the full brute-force approach which implies simply a comparison of all possible combinations.The program is run – generations are created – until a certain threshold for the given problem is found. If the variables above were parameters for a trading system the solving would stop once the profit didn’t increase, the draw down didn’t decrease, etc (meaning that there is no improvement from one generation to the next).

After doing this with the optimization of variables, people started to wonder if it could be possible to do this exact same thing with programming logic, to use a genetic approach to generate a program to carry out a certain task based on the “success” of generations of programs against a particular problem. This is when genetic programming was born. In this field we do the exact same thing as in the above mentioned genetic algorithm approach with the only difference that instead of optimizing simple parameters we optimize the logic of a program. The genetic programming approach would first generate a given number of programs with random logic – according to some initial assumptions – and then breed those who were “best fit” for the problem until we get a solution that satisfies our needs.

In the world of trading this means that in the beginning several programs are randomly generated based on some simple data which is available (like indicator and price values) and X systems are generated which would follow different trading logic sets. For example one program would just enter price after the last candle close above the weekly high while another would enter the market when the 5 period moving average was decreasing for five periods, another would use a different set of logic, etc. Then the most successful of these approaches would be bred together to arrive at a new trading logic set which would be the offspring of the first generation, this would be done time after time until we get to a point where the next generation provides no improvements over the last one. The below image shows a real example of how profit improves in future generations of trading systems.

–

What we get in the end is a trading system with optimum logic to tackle a given currency pair on a given time frame. However great care must be taken since a very complex genetic programming code which is allowed to use entry and exit logic as complex as it likes can easily curve fit the system into the randomness of the data, making the system appear extremely profitable when in reality it will not be able to achieve such success. It can also happen that the end logic will be totally impossible to understand and the inefficiencies exploited may become obscure and impossible to digest. For this reason there are several steps that need to be taken in order to prevent this from happening. The genetic programming algorithm must restrict the generations to simple exit/entry symmetric logic (same for longs as for shorts) with logic which is easy to understand and which can be easily expressed. Another very important factor is to leave some period of time (one year or two) for forward evaluation so that the logic can be tested in the future to see if it is robust enough to handle market periods for which it wasn’t optimized.

–

The end results of doing this can and are indeed quite wonderful. I have been able to code the above philosophy into Metatrader 4, using it to generate systems which have “computer designed logic” which are simple, robust and achieve to tackle varying market conditions and “unoptimized sets” with very good results. The image above shows the equity curve of one of these systems on the GBP/USD daily charts (the last 12 months on this almost 11 year test weren’t included in the initial dataset used to obtain the program’s code). Probably it will take me a few months to fully experiment with this and release the first robust genetically programmed Asirikuy system but definitely this will be a new and interesting direction for system development on the website.

If you would like to learn more about trading system development and how you can gain a true education around automated trading systems please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach to automated trading in general . I hope you enjoyed this article ! :o)

I don’t mean to sound sensational, but you keep coming up with things that I would not have imagined even 1 year ago. Somehow I keep getting surprised by the approaches and ideas that you come up with, and I read your blog entries and Asirikuy updates, finding myself saying, “really?!”

Saying that you think “outside of the box” is an understatement.

Hello Josh,

Thank you very much for your comment :o) I always try to search for new ideas to gain profit for me, my personal clients and Asirikuy members so I am always in the look to implement new methods and strategies to fit our needs for diversification and likely profitable systems. I am glad that you like my developments and that you remain a frequent blog reader. Hopefully I will have many more good ideas to create systems for Asirikuy in the future and without a doubt – with the help of Asirikuy members – the whole community will start contributing more and more towards the same goal. Thank you very much again for your comments and kind words Josh ! :o)

Best regards,

Daniel

A very exciting direction, to say the least. My gut feeling is that we cannot develop better systems this way than with manual development in terms of average yearly profit / max drawdown, but maybe we can speed up the process of developing new long term profitable systems.

I would love to read more on the theoretical and technical details.

By the way genetic algorithms could be used in finding optimal portfolio setups as well. Hm, what a perspective!

Hi Gabor,

Thank you for your comment :o) I believe the exact same thing and I have found something quite similar to what you say. The actual overall quality of the trading systems is similar or even lower than what I would personally be able to develop manually (comparing the systems with current Asirikuy experts) but the speed of development is greatly enhanced as well as the development of systems for pairs which I am not familiar with. Perhaps this is the main reason why I decided to try this approach because some pairs have very elusive inefficiencies which we can find this way.

Regarding the portfolio analysis, you are indeed right ! We could in fact use a genetic algorithm approach to optimize portfolios and find a solution to this problem. It would be great if you wanted to explore this side of the solution when you start solving this problem :o) Thank you very much for your comment !

Best Regards,

Daniel

Hi Daniel-

Great post a fascinating topic. I did some work in this area when completing my Master’s Degree in Computer Science. Unfortunately, I never had the opportunity to make any real projects out of it, but here are my brief insights.

Most AI (Artifical Intelligence) projects such as Genetic Algorithms are basically “search” problems. Given all the variables, we have a large search space and the task is to reduce the search space. Put another way, “prune” the possibility tree to a smaller search space that can be completed within a reasonable time.

Its starts with a “fitness function” which tests how your current results compare to all your prior results. In this case, i’m guessing something like Profit/Drawdown Ratio adjusted for the Pain Index. You can use your current systems results as a starting point to see whether your new systems are any better than the existing ones.

Once that’s done, the “art” aspect comes from choosing which conditions used to seed the search. That’s where knowledge of the markets come in to decide which conditions to start with since all the resulting systems will be derived from your “parental” assumptions.

I think people with first hand knowledge of the Forex markets might have some good insights based on their observations. I can’t help you there, but I can suggest the idea of “regression”. For each pair, at what times of the day that pair like to trend, versus likely to regress. This is a very powerful concept and there are some very successful commercial EA’s with long term records based on this concept. And I think it lends itself well to this type of search-based algorithm.

So those are my best ideas and I hope you can turn them into profits for the rest of us. Keep up the great work!

Chris

Hi Chris,

Thank you very much for your comments and insights :o) I am quite new to the world of genetic programming and you’ll see that what I developed is quite simple, yet effective at finding long term profitable trading systems. As you mention the initial solutions do have a lot of influence in the final outcome and much more successful systems are produced if the first approach makes at least some “trading sense”.

I’ll talk more about some of the concepts you have mentioned on future posts about this topic and I’ll certainly take into account your suggestions. Thanks again for your comment Chris !

Best Regards,

Daniel

[…] you haven’t read my previous posts on this matter I would encourage you to read this and this post where I explain the idea behind my genetic programming approach and how this can in fact lead […]

[…] systems with minors using my genetic programming framework (you can read more about it here, here and here) and what preliminary results I have been able to achieve. I will talk about the different […]