Getting data from non-Forex instruments for Forex trading in F4: Dynamically communicating with the Yahoo Finance API using curl

As I talked about in one of my recent posts, one of the largest limitations of most Forex trading platforms is the inability to obtain large amounts of historical data for non-Forex instruments that might be useful for the prediction of currency exchange rates. This means that if you want to create a system that trades the USDCAD using information from OIL, you cannot do so because your trading platform might not offer this instrument. Today I am going to talk about how I’ve overcome this limitation in the F4 framework, implementing a mechanism that allows the framework to import data from yahoo financial with the call of a very simple function. By using this increase in functionality it is now easy to build and carry out historical simulations for systems by using a wide variety of different symbols, without limitations related to the symbols available for a particular broker. Through this post I am going to talk about how I achieved this new functionality and what paths I am now going to explore with the new and powerful capabilities now available to us.

It has always been clear to me that some Forex pairs are inevitably correlated with instruments that aren’t available through most Forex brokers. For example some currency pairs might be highly correlated with commodities (as the USDCAD and OIL case I mentioned above) while others might be correlated with other types of instruments, such as the DOW Jones futures or the US T-bond interest rates. Lacking access to historical rates from these instruments is clearly limiting, because you cannot explore relationships that might prove to be important to the prediction of a pair’s exchange rate. It is clear that if there is a symbol that can forecast movements in a currency pair and you lack the ability to make this analysis you are at an inevitable disadvantage against traders who can measure and act on such forecasting power. By looking at a single currency pair – while ignoring information from instruments that might serve as leading indicators – you are trading with a blind-fold that isn’t there for those who can take a look at all the data.

23-06-2013 20-39-58

However being able to access a large historical database outside of what the Forex brokers offered had always been a distant dream for me. Not only because finding such a source seemed difficult, but because the ability to interact with such a source to seeminglessly perform both historical simulations and live trading tasks seemed daunting. This task involves the ability to somehow obtain quotes from an outside source for live trading, while at the same time having to cumbersomely pre-process and arrange historical csv files in order to load data without any issues into the platforms for back-testing. The different functionality that had to be taken into account and the possibility for tragic mistakes (read, data snooping) had always been a large enough deterrent to persuade me to avoid doing this. However – thanks to the development of our F4 trading framework at Asirikuy – I felt empowered to tackle this problem (as I mentioned on my last post about this subject).

The F4 framework – coded in C/C++ – has the big advantage that it allows me to leverage a huge repository of freely available open source libraries to carry out tasks that previously seemed impossible (especially when I was restricted to MQL4 coding). By having access to the curl open source library, I was able to see a straight path into the parsing and usage of the Yahoo financial data, in order to use it for both my live and back-testing needs. The whole process for importing data was coded in less than 100 lines and is carried out in the exact same manner for both historical and live testing. This means that the process is inherently robust, as when live trading I am following the exact same process as when back-testing. The image above highlights the steps I carry out each time an algorithmic strategy that uses this new functionality is called.

The first step I carry out is simply to make an HTTP get request to the Yahoo financial service, using the instrument ticker specified by the user and the desired starting date. The requesting of data is always restricted to the framework’s current trading time, so in this sense it is impossible to get data that is “future” to the date available to the framework (no snooping). When importing the data I also make sure that the current trading day’s data is modified so that all values are equal to the open, so that the possibility of snooping due to time differences with the yahoo data is reduced to zero. After importing the data into a rates array within the F4 framework it is then ready for immediate use by the system, using the exact same functions used to handle rates arrays. The user can calculate indicators on this data, get OHLC information, use the data for machine learning, etc. The syncing of the data can vary on an index-to-index basis on the arrays (because holidays and trading days might vary) but the importing process guarantees that no snooping is possible (so you cannot possibly peak into the future when back-testing). One of the main advantages is that this process happens in the exact same way for back-testing and live trading.

23-06-2013 20-48-17

Another huge advantage of this process is the easiness with which the user can carry out this process. In the F4 framework code this only requires one simple function code of the form addNewDailyRates(ticker, initialTime, assignedIndex), where the user only needs to specify a few key pieces of information, which are the ticker, the time from which the data array is to be filled and the index assigned to the new rates. After this the new index can be used to access all the information from this new rates array. For example if I want to get the CBOE Interest Rate 10-Year T-Note for the past 200 days, it is as simple as calling: addNewDailyRates(“^TNX”, openTime(200), 2). Tell me how you would do that in MT4! ;o)

The implementation of this functionality has just been finished but it will be available to all Asirikuy members from next week-end’s update :o) If you would like to learn more about the F4 framework and how you too can learn to build systems with a powerful C/C++ framework  please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading in general . I hope you enjoyed this article ! :o)




Print Friendly, PDF & Email
You can leave a response, or trackback from your own site.

Leave a Reply

Subscribe to RSS Feed Follow me on Twitter!
Show Buttons
Hide Buttons