Mechanical Forex

When you browse for freely available data for the Forex market online you will often find that this data is in the obscure HST binary format for the MT4 platform. Using data that is in this format is quite painful as you often have to convert it into a format you can work with (like a csv) using the history center functionality from the Metatrader 4 platform. However doing this is highly inconvenient since the MT4 program is a bloated application that does not allow for the batch processing of history files and also has some serious memory limitations due to the nature of its code. Today we are going to learn how to easily perform the conversion from hst to csv without having to go into the pain of using the Metatrader 4 platform. We are also going to learn some surprising things about what the MT4 platform actually does when you import and then export an hst file.

–

#!/usr/bin/python
# Coded by Daniel Fernandez
# mechanicalForex.com, asirikuy.com 2015

import struct
from time import sleep
import time
import pandas as pd
import datetime
import argparse

HEADER_SIZE = 148
OLD_FILE_STRUCTURE_SIZE = 44
NEW_FILE_STRUCTURE_SIZE = 60

def main():

    parser = argparse.ArgumentParser()
    parser.add_argument('-f', '--filename')
    parser.add_argument('-ty', '--filetype')
    args = parser.parse_args()

    filename = args.filename
    filetype = args.filetype
    
    if filename == None:
        print "Enter a valid filename (-f)"
        quit()
        
    if filetype != "new" and filetype != "old":
        print "Enter a valid filetype (valid options are old and new)"
        quit()

    read = 0
    openTime = []
    openPrice = []
    lowPrice = []
    highPrice = []
    closePrice = []
    volume = []

    with open(filename, 'rb') as f:
        while True:
            
            if read >= HEADER_SIZE:
            
                if filetype == "old":
                    buf = f.read(OLD_FILE_STRUCTURE_SIZE)
                    read += OLD_FILE_STRUCTURE_SIZE        
                         
                if filetype == "new":
                    buf = f.read(NEW_FILE_STRUCTURE_SIZE)
                    read += NEW_FILE_STRUCTURE_SIZE
                    
                if not buf:
                    break
                    
                if filetype == "old":
                    bar = struct.unpack("<iddddd", buf)
                    openTime.append(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(bar[0])))
                    openPrice.append(bar[1])
                    highPrice.append(bar[3])
                    lowPrice.append(bar[2])
                    closePrice.append(bar[4])
                    volume.append(bar[5])  
                if filetype == "new":
                    bar = struct.unpack("<Qddddqiq", buf)
                    openTime.append(time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(bar[0])))
                    openPrice.append(bar[1])
                    highPrice.append(bar[2])
                    lowPrice.append(bar[3])
                    closePrice.append(bar[4])
                    volume.append(bar[5])  
                                              
            else:           
                buf = f.read(HEADER_SIZE)
                read += HEADER_SIZE
                
    data = {'0_openTime':openTime, '1_open':openPrice,'2_high':highPrice,'3_low':lowPrice,'4_close':closePrice,'5_volume':volume}
     
    result = pd.DataFrame.from_dict(data)
    result = result.set_index('0_openTime')
    print result
     
    result.to_csv(filename+'_.csv', header = False)   
        
        
##################################
###           MAIN           ####
##################################

if __name__ == "__main__": main()

–

I was quite surprised when a few days ago I wanted to see the differences between my historical data and the 1M historical data from FXDD (which is available here for free) but I could not find a simple way to convert hst to csv files online. After suffering from several MT4 platform crashes and taking more than 5 minutes to convert a single file I was done with it and decided to code my own implementation for the conversion of the MT4. This is how I came up with the script you see above which you can easily use to make this conversion whenever you need to. I will now show you how to use this script and talk a bit about the hst file structure.

The hst binary file format is actually not a single file format. There are two versions of these binary type (see here) depending on whether the data was generated for an old MT4 platform or a new MT4 platform. Both versions retain a 148 byte header but the old format has the OHLC data within a 44 byte structure while the new data has it in a 60 byte structure. The difference is because the new data contains additional fields for spread and real_volume which are quite handy if you want to have more accuracy in the data that you store. When using the script you can convert any hst file by pointing at the file and whether the file is in the “new” or “old” format. For example to convert the 1M data from FXDD I used the command “python hst_to_csv_converter.py -f “EURUSD.hst” -ty old -tf 1″. Since this FXDD data comes in the old format I specified the keyword “old”. After you do this the script will generate a pandas dataframe which will then go into a csv file with the default pandas formatting.

–

–

It is easy to see how this script can be easily used to process an entire batch of hst files to completely rid yourself of the MT4 platform for the purpose of hst file reading and exporting. Once you have this script you have no need to Import/Export data in MT4 but you simply need the hst file and you can obtain a clean csv with the whole data series in human readable format. The data you get is exactly the same as the data you would export from the MT4 history center (although some differences in formatting are indeed present). The script also converts the data into a pandas dataframe before exporting so if you know your way around pandas you can already use this loaded data for anything you want to do without the need to save back to a csv file.

Once you have your data in the above csv format you can also easily analyse it using something like R.  The code below loads your data into an R dataframe using the proper datetime format, then changes the column names to match OHLCV, shows you the head of your data and finally performs some simple series plots using the quantmod library. Once you have your data in R you can also perform all sorts of statistical analysis, you can evaluate things as entropy, randomness, etc.

–

library(quantmod)

f <- function(x, format) {
  as.POSIXct(paste0(as.character(x)), format = format)
}

fxdatatemp <- read.zoo("pathToFile/EURUSD.hst_.csv", sep = ",",format="%Y-%m-%d %H:%M", header=FALSE,index.column=1,FUN=f)
eurusd<- xts(fxdatatemp)
colnames(eurusd) <- c('open','high','low','close','volume')
head(eurusd)
chartSeries(eurusd) 
barChart(eurusd,theme='white.mono',bar.type='hlc')

–

–

Certainly it’s a pleasure when you can process your data as you wish with a wide variety of different tools at your disposal. If you would like to learn more about loading data, comparing data and performing other data analysis tasks using R and python for Forex trading please consider joining Asirikuy.com, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach towards automated trading.