Downloading and aggregating high frequency trading data from Bovespa

Marcelo Perlin

2016-07-21

Recently Bovespa, the Brazilian financial exchange company, has released the access to its internal ftp site. In this address one can find several information regarding the Brazilian financial system, including datasets with high frequency (tick by tick) trading data for three different markets: equity, options and BMF.

Downloading and processing these files, however, can be exausting. The dataset is composed of zip files with the whole trading data, separated by day and market. These files are huge in size and processing or aggregating them in a usefull manner requires specific knowledge for the structure of the dataset.

The package GetHFData make is easy to access this dataset directly by allowing the easy importation and aggregations of it. Based on this package the user can:

In the next example we will only use a local file from the package. Given the size of the files in the ftp and the CHECK process of CRAN, it makes sense to keep this vignette compact and fast to run.

Reading trading data from local file (1 date)

Lets assume you want high frequency trading data for options in a given date (2015-11-26). This file could be downloaded from the ftp using function ghfd_download_file, but it is already available locally in the package.

The first step is to check the available tickers in the zip file:

library(GetHFData)

out.file <- system.file("extdata", 'NEG_OPCOES_20151126.zip', package = "GetHFData")
tickers.out <- ghfd_get_available_tickers_from_file(out.file)
print(tickers.out[1:10]) # show only 10
## 
## PETRL80  PETRL9 VALEL14 VALEL43 PETRL70 PETRX80 VALEA43 ITUBA14 VALEL12 
##    2882     985     754     679     514     507     478     415     383 
##  ITUBL1 
##     270

In tickers.out one can find the symbols available in the file and also the number of trades for each. Now, lets take the 3 most traded instruments in that day and check the result of the import process:

my.assets <- names(tickers.out[1:3]) # ticker to find in zip file

start.time <- '10:00:00' # defines first time period of day
last.time <- '17:00:00'  # defines last time period of day

my.df <- ghfd_read_file(out.file,
                        my.assets = my.assets,
                        first.time = '10:00:00',
                        last.time = '17:00:00',
                        type.output = 'raw',
                        agg.diff = '15 min')
##  - Imported  18303 lines, 771 unique tickers
##    -> Processing file - Found 4621 lines, 3 unique tickers

Let’s see the first part of the imported dataframe.

head(my.df)
## # A tibble: 6 x 7
##   SessionDate InstrumentSymbol TradePrice TradedQuantity    Tradetime
##        <date>            <chr>      <dbl>          <dbl>        <chr>
## 1  2015-11-26          PETRL80       0.43            600 10:00:17.430
## 2  2015-11-26          PETRL80       0.44          11300 10:00:28.451
## 3  2015-11-26          PETRL80       0.45           2500 10:00:32.160
## 4  2015-11-26          PETRL80       0.45            500 10:00:34.676
## 5  2015-11-26          PETRL80       0.45           5000 10:00:36.274
## 6  2015-11-26          PETRL80       0.45           1000 10:00:53.221
## # ... with 2 more variables: TradeDateTime <time>, TradeSign <dbl>

The columns names are self explanatory:

names(my.df)
## [1] "SessionDate"      "InstrumentSymbol" "TradePrice"      
## [4] "TradedQuantity"   "Tradetime"        "TradeDateTime"   
## [7] "TradeSign"

Now lets plot the prices of all instruments:

library(ggplot2)
 
p <- ggplot(my.df, aes(x = TradeDateTime, y = TradePrice, color = InstrumentSymbol))
p <-  p + geom_line()
print(p)

As we can see, this was a fairly stable day for the price of these option contracts.

Downloading and reading trading data for several dates

In the last example we only used one date. The package GetHDData also allows for a batch download and processing of files using starting and ending dates. In this vignette we are not running the code given the large size of the downloaded files. You should try the next example in your own computer (just copy, paste and run the code in R).

In this example we will download files from the ftp for PETR4, an equity contract traded in Bovespa. The data will be processed, resulting in financial data aggregated by 15 minutes.

library(GetHFData)

first.time <- '11:00:00'
last.time <- '17:00:00'

first.date <- as.Date('2015-11-01')
last.date <- as.Date('2015-11-10')
type.output <- 'agg'
agg.diff <- '15 min'

my.assets <- 'PETR4'
type.market = 'equity'

df.out <- ghfd_get_HF_data(my.assets =my.assets,
                           type.market = type.market,
                           first.date = first.date,
                           last.date = last.date,
                           first.time = first.time,
                           last.time = last.time,
                           type.output = type.output,
                           agg.diff = agg.diff,
                           clean.files = F)