rdwd: climate data from the German Weather Service

Berry Boessenkool, berry-b@gmx.de

2018-11-26

Vignette Rmd source code (Not on CRAN to reduce load on DWD server through daily new builds and checks of the vignette)

Interactive map vignette

Intro

The R package rdwd contains code to select, download and read weather data from measuring stations across Germany. The German Weather Service (Deutscher Wetterdienst, DWD) provides over 228 thousand datasets with weather observations through the FTP server online at

ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate.

For data interpolated onto a 1 km raster, including radar data up to the last hour, see ftp://ftp-cdc.dwd.de/pub/CDC/grids_germany/. Management of these (very large!) datasets is not yet included in rdwd. If you want it to be supported, send me a message to boost it up my priority list.

For further details, please consult the DWD FTP server documentation

Package structure

To use the observation datasets, rdwd has been designed to mainly do 3 things:

selectDWD uses the result from indexFTP which recursively lists all the files on an FTP-server (using RCurl::getURL). As this is time consuming, the result is stored in the package dataset fileIndex. From this, metaIndex and geoIndex are derived.

TOC

Package installation

install.packages("rdwd")
# get the latest development version from github:
berryFunctions::instGit("brry/rdwd") 
# For full usage, as needed in indexFTP and metaDWD(..., current=TRUE):
install.packages("RCurl") # is only suggested, not mandatory dependency

On Linux, instead of the last line above, use in the terminal (with lowercase rcurl):

sudo apt-get install r-cran-rcurl

If direct installation from CRAN doesn’t work, your R version might be too old. In that case it is really recommendable to update R. If you can’t update R, try installing from source (github) via instGit as mentioned above. If that’s not possible either, you might be able to source some functions from the package zip folder

Vectorize(source)(dir("path/you_unzipped_to/rdwd-master/R", full=T))

TOC

Basic usage

library(rdwd)
link <- selectDWD("Potsdam", res="daily", var="kl", per="recent")
file <- dataDWD(link, read=FALSE, dir=tempdir(), quiet=TRUE)
# tempdir is only for CRAN vignette checks. In real life, use a real folder.
clim <- readDWD(file)

str(clim)
## 'data.frame':    550 obs. of  19 variables:
##  $ STATIONS_ID: int  3987 3987 3987 3987 3987 3987 3987 3987 3987 3987 ...
##  $ MESS_DATUM : POSIXct, format: "2017-05-25" "2017-05-26" ...
##  $ QN_3       : int  10 10 10 10 10 10 10 10 10 10 ...
##  $ FX         : num  9.7 8.7 8.8 12 7.3 20.3 15.6 9.8 8.2 8.8 ...
##  $ FM         : num  3.6 3.6 3.4 4.2 2.7 4.1 6 4.1 2.6 3.9 ...
##  $ QN_4       : int  3 3 3 3 3 3 3 3 3 3 ...
##  $ RSK        : num  0 0 0 0 0 0.5 0 0 0 3.8 ...
##  $ RSKF       : int  0 0 0 0 0 6 0 0 0 6 ...
##  $ SDK        : num  7.03 9.13 15.78 12.4 12.35 ...
##  $ SHK_TAG    : int  0 0 0 0 0 0 0 0 0 0 ...
##  $ NM         : num  5 3.7 0 3.4 4 5.3 5.1 2 1.9 6.3 ...
##  $ VPM        : num  11.4 13.8 11.9 13.2 16.5 18.4 12.7 8.9 8.8 13.8 ...
##  $ PM         : num  1009 1009 1011 1007 1002 ...
##  $ TMK        : num  13.7 17.2 20.2 23.2 23.8 22.3 18.7 15.7 18.6 17.2 ...
##  $ UPM        : num  73.9 73.3 55 48.6 58.4 ...
##  $ TXK        : num  18.9 24.9 27.9 32.3 30.6 31.6 24.6 23.6 26.2 22.7 ...
##  $ TNK        : num  8.3 11.7 12.2 13.2 17.2 18.1 11.2 8.7 10.2 12 ...
##  $ TGK        : num  4.7 8.2 7.8 10.3 13.9 15.7 8.8 6.4 5.8 8.7 ...
##  $ eor        : Factor w/ 1 level "eor": 1 1 1 1 1 1 1 1 1 1 ...

TOC

Plotting examples

Recent temperature time series:

par(mar=c(4,4,2,0.5), mgp=c(2.7, 0.8, 0), cex=0.8)
plot(clim[,c(2,4)], type="l", xaxt="n", las=1, main="Daily temp Potsdam")
berryFunctions::monthAxis(ym=TRUE)   ;   abline(h=0)
mtext("Source: Deutscher Wetterdienst", adj=-0.1, line=0.5, font=3)

Long term climate graph:

link <- selectDWD("Goettingen", res="monthly", var="kl", per="h")
clim <- dataDWD(link, quiet=TRUE)
clim$month <- substr(clim$MESS_DATUM_BEGINN,5,6)
temp <- tapply(clim$MO_TT, clim$month, mean, na.rm=TRUE)
prec <- tapply(clim$MO_RR, clim$month, mean, na.rm=TRUE)
library(berryFunctions)
headtail(clim[!is.na(clim$MO_TT)&!is.na(clim$MO_RR),])
##      STATIONS_ID MESS_DATUM_BEGINN MESS_DATUM_ENDE QN_4 MO_N MO_TT MO_TX
## 1           1691          18570101        18570131    5   NA -1.00    NA
## 1932        1691          20171201        20171231    3 7.46  3.78  5.89
##      MO_TN MO_FK MX_TX MX_FX MX_TN MO_SD_S QN_6 MO_RR MX_RS eor month
## 1       NA    NA    NA    NA    NA      NA    5  15.0    NA eor    01
## 1932  1.26  2.77    13  18.4  -2.9    16.3    3  48.8   7.7 eor    12
# as of 2018-03, there are mostly NAs in MO_RR in many stations.
# A message has been sent to DWD.
climateGraph(temp, prec, main="Goettingen 1857:1946")
mtext("Source: Deutscher Wetterdienst", adj=-0.05, line=2.8, font=3)

TOC

Station selection

Weather stations can be selected geographically with the interactive map. All stations within a certain radius around a given lat-long position can be obtained with nearbyStations.

The DWD station IDs can be obtained from station names with

findID("Potsdam")
## Potsdam 
##    3987
findID("Koeln", exactmatch=FALSE)
## Warning: source -> withVisible -> eval -> eval -> createBerrysVignettes -
## > rmarkdown::render -> knitr::knit -> call_block -> block_exec -> in_dir -
## > evaluate -> evaluate::evaluate -> evaluate_call -> timing_fn -> handle -
## > findID: ID determined from name 'Koeln' has 4 elements (2665, 2666, 2667,
## 2968).
##               Koeln-Bonn Koeln-Botanischer Garten           Koeln-Porz-Eil 
##                     2667                     2665                     2666 
##          Koeln-Stammheim 
##                     2968

TOC

Available files

File selection by station name/id and folder happens with selectDWD. It needs an index of all the available files on the server. The package contains such an index (fileIndex) that is updated (at least) with each CRAN release of the package. The selectDWD documentation contains an overview of the FTP folder structure.

If you find the file index to be outdated (Error in download.file … : cannot open URL), please let me know and I will update it. Meanwhile, use current=TRUE in selectDWD:

# all files at a given path, with current file index (RCurl required):
links <- selectDWD(res="monthly", var="more_precip", per="hist", current=TRUE)

fileIndex is created with the function indexFTP used in the last section of rdwd-package.R.

# recursively list files on the FTP-server:
files <- indexFTP("hourly/sun") # use dir="some_path" to save the output elsewhere
berryFunctions::headtail(files, 5, na=TRUE)

# indexFTP uses a folder for resumed indexing after getting banned:
gridindex <- indexFTP("radolan","ftp://ftp-cdc.dwd.de/pub/CDC/grids_germany/hourly")
gridindex <- indexFTP(gridindex,"ftp://ftp-cdc.dwd.de/pub/CDC/grids_germany/hourly", sleep=1)

# with other FTP servers, this should also work...
funet <- indexFTP(base="ftp.funet.fi/pub/standards/w3/TR/xhtml11/", folder="")
p <- RCurl::getURL("ftp.funet.fi/pub/standards/w3/TR/xhtml11/",
                       verbose=T, ftp.use.epsv=TRUE, dirlistonly=TRUE)

TOC

File selection

selectDWD is designed to be very flexible:

# inputs can be vectorized, and period can be abbreviated:
selectDWD(c("Potsdam","Wuerzburg"), res="hourly", var="sun", per="hist")
## [[1]]
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_03987_18930101_20171231_hist.zip"
## 
## [[2]]
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/hourly/sun/historical/ stundenwerte_SD_05705_19510101_20171231_hist.zip"
# Time period can be doubled to get both filenames:
selectDWD("Potsdam", res="daily", var="kl", per="rh", outvec=TRUE)
## [1] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/recent/ tageswerte_KL_03987_akt.zip"                       
## [2] "ftp://ftp-cdc.dwd.de/pub/CDC/observations_germany/climate/daily/kl/historical/ tageswerte_KL_03987_18930101_20171231_hist.zip"

There may be a differing number of available files for several stations across all folders. That’s why the default outvec is FALSE.

lapply(selectDWD(id=c(3467,5116)), substr, 58, 1e4)
## Warning: source -> withVisible -> eval -> eval -> createBerrysVignettes -
## > rmarkdown::render -> knitr::knit -> call_block -> block_exec -> in_dir -
## > evaluate -> evaluate::evaluate -> evaluate_call -> timing_fn -> handle -
## > lapply -> selectDWD: in file index 'fileIndex', there are 6 files with ID
## 3467.
## Warning: source -> withVisible -> eval -> eval -> createBerrysVignettes -
## > rmarkdown::render -> knitr::knit -> call_block -> block_exec -> in_dir -
## > evaluate -> evaluate::evaluate -> evaluate_call -> timing_fn -> handle -
## > lapply -> selectDWD: in file index 'fileIndex', there are 3 files with ID
## 5116.
## [[1]]
## [1] "/annual/more_precip/historical/jahreswerte_RR_03467_19940101_20171231_hist.zip" 
## [2] "/annual/more_precip/recent/jahreswerte_RR_03467_akt.zip"                        
## [3] "/daily/more_precip/historical/tageswerte_RR_03467_19930601_20171231_hist.zip"   
## [4] "/daily/more_precip/recent/tageswerte_RR_03467_akt.zip"                          
## [5] "/monthly/more_precip/historical/monatswerte_RR_03467_19930601_20171231_hist.zip"
## [6] "/monthly/more_precip/recent/monatswerte_RR_03467_akt.zip"                       
## 
## [[2]]
## [1] "/annual/more_precip/historical/jahreswerte_RR_05116_19930101_20051231_hist.zip" 
## [2] "/daily/more_precip/historical/tageswerte_RR_05116_19920701_20061231_hist.zip"   
## [3] "/monthly/more_precip/historical/monatswerte_RR_05116_19920701_20061231_hist.zip"

TOC

Metadata

selectDWD also uses a complete data.frame with meta information, metaIndex (derived from the “Beschreibung” files in fileIndex).

# All metadata at all folders:
data(metaIndex)
str(metaIndex, vec.len=2)
## 'data.frame':    76772 obs. of  12 variables:
##  $ Stations_id  : int  1 1 1 1 1 ...
##  $ von_datum    : int  18910101 18910101 18910101 18910101 19120101 ...
##  $ bis_datum    : int  19860630 19860630 19860630 19860630 19860630 ...
##  $ Stationshoehe: num  478 478 478 478 478 ...
##  $ geoBreite    : num  47.8 47.8 ...
##  $ geoLaenge    : num  8.85 8.85 ...
##  $ Stationsname : chr  "Aach" "Aach" ...
##  $ Bundesland   : chr  "Baden-Wuerttemberg" "Baden-Wuerttemberg" ...
##  $ res          : chr  "annual" "annual" ...
##  $ var          : chr  "more_precip" "more_precip" ...
##  $ per          : chr  "historical" "recent" ...
##  $ hasfile      : logi  TRUE FALSE TRUE ...
View(data.frame(sort(unique(rdwd:::metaIndex$Stationsname)))) # ca 6k entries

dataDWD can download (and readDWD can correctly read) such a data.frame from any folder on the FTP server:

# file with station metadata for a given path:
m_link <- selectDWD(res="monthly", var="more_precip", per="hist", meta=TRUE)
substr(m_link, 50, 1e4) # (Monatswerte = monthly values, Beschreibung = description)
## [1] "/climate/monthly/more_precip/historical/RR_Monatswerte_Beschreibung_Stationen.txt"
meta_monthly_rain <- dataDWD(m_link, dir=tdir) # not executed in vignette creation
str(meta_monthly_rain)

Meta files may list stations for which there are actually no files. For example: Tucheim (5116) is listed in the metadata at …/monthly/more_precip/recent/RR_Monatwerte_Beschreibung_Stationen.txt, but actually has no file in that folder (only in …/monthly/more_precip/historical). These refer to nonpublic datasets (The DWD cannot publish all datasets because of copyright restrictions). To request those, please contact klima.vertrieb@dwd.de.

TOC

Any feedback on this package (or this vignette) is very welcome via github or berry-b@gmx.de!