One of the problems of not having a centralized exchange for trading in the Forex market is the fact that each particular liquidity provider can create its own data based on what is most convenient to them. This not only means that there can be significant differences between different feeds regarding the data itself but it also means that data can be represented using time stamps from different time zones. This means that you can have back-testing data that comes from a provider using a GMT offset of +2 and then have to live trade on a broker using GMT -5. Due to the fact that GMT offsets and DST changes are rarely described in detail by the brokers and data providers themselves it becomes imperative to find some manner to find out what the GMT offset and DST changes for the data actually are. Today I will show you how you can use a simple python script to carry out this task.
–
#!/usr/bin/python
#Automatic GMT/DST detection
#Programmed by Daniel Fernandez 2015
#http://mechanicalforex.com
#https://asirikuy.com
import pandas as pd
import numpy as np
def main():
historyFilePath = "pathToData.csv"
#note that this assumes that the timestamps within the data are in d/m/y format.
#if you have a different format changes the pd.read_csv command appropiately
df = pd.read_csv(historyFilePath, index_col=0, engine='python',parse_dates=True, dayfirst=True, header=None, names=["Time","Open","High","Low","Close","Volume"])
hour = df.index.hour
dayofweek = df.index.dayofweek
dayofmonth = df.index.day
month = df.index.month
year = df.index.year
selector_nondst = ((4==dayofweek) & (dayofmonth<7) & (month==2) & (year==2015))
selector_dst = ((4==dayofweek) & (dayofmonth<7) & (month==6) & (year==2015))
df_nondst = df[selector_nondst]
df_dst = df[selector_dst]
df_nondst['Range'] = df_nondst['High']-df_nondst['Low']
df_dst['Range'] = df_dst['High']-df_dst['Low']
print "----------------------"
print "non-DST NFP happens at hour {}".format(df_nondst['Range'].idxmax().hour)
print "DST NFP happens at hour {}".format(df_dst['Range'].idxmax().hour)
print "non-DST GMT shift is: {}".format(df_nondst['Range'].idxmax().hour-13)
print "DST GMT shift is: {}".format(df_dst['Range'].idxmax().hour-12)
print "----------------------"
##################################
### MAIN ####
##################################
if __name__ == "__main__": main()
–
Why is it so important to know? Well, in the Forex market you have trading 24/5 and this means that for the overall market to say open there must actually be an overlap of different markets opening and closing for trading around the world during the day. The Forex market is well known for its marked hourly based volatility cycles that depend on the different opening and closing times for markets around the world. This behavior almost warrants that inefficiencies will be associated with specific times and therefore implies that your trading can go heavily astray if you intended to trade on one hour but end up trading on another because you did not properly know the times within your simulations. Not having proper knowledge about what your broker’s GMT shift is or whether or not this shift changes with DST can turn a strategy that would have been profitable into a complete disaster.
How can we find then? This has often been done manually through cumbersome comparison steps that involve having a feed with a presently known GMT offset and DST change and then searching for a matching candle within your testing feed. If you have a known feed that is presently GMT 0 and you can locate a matching candle on your testing feed with a timestamp shifted by 1 hour forward then you know your GMT offset is +1 that time of the year, you can then go back to a time where DST is different and you can get the same answer for the DST valid period. However this process can be automated using a common Forex reference, the non-farm payrolls news release.
–
–
The NFP news release is very unique because it has two very useful characteristics. The first is that this release always happens at the same time (8:30 US EST, first business Friday of each month) and the second is that this release always causes a significant increase in volatility from the hours that come before it. Using this information I created the script above – works with 1H data – that searches for the NFP release in February and in June 2015 and then compares the times found within the data with the expected time in the US EST timezone in order to calculate the actual time. The NFP is easy to locate because it’s the large volatility peak that happens within the first Friday of the month and knowing that this is always related the US EST timezone allows us to easily pinpoint the actual timezone of the real data. Running this script on my data – results above – it easily concludes that my data is GMT +1/+2 which is correct.
It is also worth noting that the above script provides a basic mechanism to make this comparison assuming that the GMT and DST of your data do not change through the years. Some brokers in the Forex market have so poorly constructed data that it presents arbitrary changes in the GMT/DST offsets across the years and therefore it becomes necessary to change the year within the script to perform this analysis on every trading year from your data. I generally do this with new data sets in order to really find out if the GMT/DST of the data is really consistent through the years. For example in the case of an Alpari data set I analysed I found out that the GMT was +1/+2 up until 2011 and then it changed to GMT +2/+3. If I hadn’t carried out this year by year analysis and assumed that the entire data was either GMT +1/+2 or GMT +2/+3 I would have made a huge mistake. Once you know which years have which GMT/DST offsets you can correct the data to make it homogeneous or even match the GMT/DST offset your desire (which is what I did).
–
–
If you would like to learn more about working with Forex data and how you too can work with a programming framework that performs automatic GMT/DST corrections so that you always back-test and live trade using the same time stamp structure please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.
