Mechanical Forex

The past year was a very interesting time for me in Forex trading. In 2014 I had already started moving away from the traditional algorithmic retail trader notion of only trading a few manually designed automated trading strategies and I started developing what would become the core of the system mining technologies we currently use to generate trading systems. However it wasn’t until mid 2014 and 2015 that our mining effort exploded and we were able to find thousands of trading strategies that we now trade on live Forex accounts. On today’s post I want to talk about some of the lessons we have learned from attempting to implement this massive mining and system effort and what I see within the future for our development team and trading community.

–

–

In the beginning our GPU based mining was really shy. It came from developments made in price action based mining – what you can now get for free as OpenKantu on this blog – and implemented some large improvements that were aimed at generating better systems faster instead of making prettier graphs (as OpenKantu is more oriented at doing small research experiments rather than full-fledged massive mining). Our GPU mining is 1000x faster than the CPU based mining we had before and can use a variety of different trade management mechanisms not present within the OpenKantu software.

Our system building effort really started with the creation of cloud based mining which allowed all of us to contribute our GPU time to the same system finding effort. Doing this we were able to start validating systems through data-mining bias analysis and started creating what is now a trade system repository with more than 3500 strategies. The first system came in June 2014 and it took us nearly two months to add the following 6 systems. There were lots of initial problems at this stage, we had a very small amount of GPU processing power and the mining process was not entirely automated. The manual steps involved in some parts of the process – mainly the part that involved system correlation analysis and repository addition – made the process quite slow although this was not a limiting factor given the slowness of the mining operation at this point (and we were already mining millions of potential systems per day!).

–

–

At the start of 2015 we already had 52 systems and we were well in our way to formally establish a solid methodology for automated system creation within a cloud centered effort. Mining continued to evolve and pick up speed as we were able to get our hands on more GPU mining power, meaning that by July 2015 we already had more than 300 trading systems available within our repository. During 2015 we also added 4 new pairs to our mining effort which implied that portfolio diversity increased quite substantially. Our mining finally exploded in the second half of 2015 thanks to the entire automation of the entire mining process. The implementation of entirely automated mining space generation (automating the answer to the “what to mine now?” question) and the automation of the entire system correlation analysis and repository addition steps meant that we were no longer bound by my ability to process mining results – at least partially – by hand. These changes created a mining boom that generated many more systems and by January 2016 our repository had already gone above the 3000 system mark.

Of course an entirely different issue was to trade those 3000 systems on real accounts at the same time. In order to do this we implemented an order execution engine based on the Oanda REST API. We created stripped down software that was streamlined to execute just the mined strategies and we implemented a powerful virtual order manager based on powerful dedicated server based mysql implementations. Currently we can execute our 3500 strategies across more than 70 live trading accounts in less than one minute. Another reason why we decided to use Oanda was their ability to fraction lot sizes down to 1 USD sizes, meaning that we could effectively trade a 3500 system portfolio on an account with less than 2000 USD with a total risk of 6%. This is amazing if you think about it as it means that on an equally weight portfolio we currently trade at a  0.0017% risk per trade. Talk about fine risk management!

–

–

From this journey in mining we have learned many important lessons. The first is that it always pays to reduce your computational costs, if you can optimize your code and reduce your time by 1% per system simulation then that can mean days when you’re mining billions of strategies. If you have to go through a lot of effort to develop something much faster – as we did with GPU technology – then by all means do so, the time you spend may seem like a lot at first but what are a few months if you can then save years. We also learned that it is critically important to be able to centralize your efforts so that you all focus on the same thing. If everyone uses mining software to do their own thing then the effort is dispersed, to reach our current portfolio status cloud based mining where everyone could contribute to the same thing was essential. Along this same line of thought it is also tremendously important to make sure you automate everything, manual steps introduce bottlenecks that you won’t want when you’re building something with the potential to become massive.

On the execution front we also learned many lessons from trading massive arrays of strategies. We learned that broker-side limitations are something very important to account for when you’re trading so many things – for example how many rate requests per second you can make – and that things such as rate requesting and bid/ask loading have to be tremendously optimized to be able to trade a wide variety of systems in a manner that is fast enough. We also learned a lot about MySQL transaction limitations and database optimization since there were some substantial problems when handling the transaction volume you expect from +3500 systems. We currently have an average of around 600 new trades per day and eagerly hoping to expand to higher volumes as our mining becomes more and more powerful.

–

–

Our mining currently continue to increase in power and our horizons continue to expand. Presently we have implemented new trailing stop mechanisms and continue to look for additional Forex strategies worth mining but there are certainly some additional avenues that we’re seeking to explore (left for a future post!). We are also repeating this same process with machine learning systems, where we are now at the 98 system mark and hope to continue evolving as we gather more momentum and mining power. Increasing our mining capacity in machine learning mining is much harder – another future post will explain why this is the case – but we are now looking into some massively parallel options that may help us tremendously.

All in all we are now expanding into unexplored territory for the average retail trader, with a capacity to trade thousands of strategies and mine hundreds of new systems per week there is simply nothing in the retail sector that compares to what we can do now (at least that I know of). If you would like to learn more about system mining and how you too can learn how to generate trading strategies please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general.