--- title: "Getting Started with data4health: A Practical Workflow for Health Data Wrangling" output: rmarkdown::html_vignette: toc: true vignette: > %\VignetteIndexEntry{Getting Started with data4health: A Practical Workflow for Health Data Wrangling} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` # Overview The **data4health** package streamlines the typical epidemiological data workflow: loading raw surveillance or clinical files, cleaning and standardising them, filtering the cases of interest, aggregating case counts over space and time, and finally saving the results for downstream analysis or reporting. It is designed to help users with the most common challenges encountered during health data wrangling, such as inconsistent column names, missing values, fragmented reporting periods, and the need for spatiotemporal aggregation. The **data4health** package is designed to work in tandem with other packages of the 4health suite: * **clim4health**, which focuses on climate data, * **socio4health**, which focuses on socio-economic data, * **land4health**, which focuses on land-use data, * **drone4health**, which focuses on drone data, * **cube4health**, which helps to create datacubes. Together, these packages support the integration of environmental, social, and health data into a unified analytical workflow. More information about the toolkit can be found at the [HARMONIZE website](https://harmonize-tools.github.io/). This vignette walks you through a complete worked example using synthetic dengue surveillance data to provide an overview of the **data4health** package workflow and its core functions. ```{r setup} library(data4health) ``` ## Who is this vignette for? This vignette is written for: - Public health professionals working with surveillance data - Data analysts supporting health departments - Researchers handling epidemiological datasets - Data scientists who are new to health data structures No prior knowledge of epidemiology is required, but basic familiarity with R and data frames is assumed. ## Data requirements For the workflows shown in this package, the input data must be organized as a line list dataset. A “line list” is a standard format in public health surveillance. It is simply a spreadsheet where each row represents one individual case (for example, one patient with dengue), and each column contains information about that individual (such as age, sex, location, and date of symptom onset). In this vignette we work with a synthetic dataset that was simulated to mirror health data that public health authorities collect when monitoring infectious diseases: ```{r inspect_data} head(example_dataset) ``` The dataset includes the following variables: * NAME_GEO: Name of the geographic area (e.g., city or district) where the case was reported * CODE_GEO: Numeric geographic code corresponding to the area * LOCALCOD: Local administrative code * DATE_BRTHD: Date of birth of the individual * DATE_INI: Date of symptom onset (when the illness began) * SEXO: Sex of the individual (M = Male, F = Female) * DENGUE_TYPE: Clinical classification of dengue severity (coded numerically) * HOSPITALISED: Whether the patient was hospitalised (S = Yes, N = No) * RESULT_LABORATORY: Laboratory test result (Positive, Negative, or Pending) * NOTES: Free-text notes (may contain additional comments) * SEROTYPE: Dengue virus serotype (when available) --- # The data4health workflow ``` d4h_load() → d4h_clean() → d4h_filter() → d4h_aggregate() → d4h_save() ``` Each step produces a plain `data.frame` that you can inspect, subset, or pass directly into the next function. You can choose to use the full pipeline or just choose the functions that fit your needs. The full workflow can be executed in one go, or you can choose to execute each step separately and inspect the intermediate results before moving on to the next step. This flexibility allows you to tailor the workflow to your specific needs and data. ## 1. Loading data with `d4h_load()` `d4h_load()` accepts a path to either a single file or a character vector of paths and binds the results into one dataframe. Supported formats include `.csv`, `.rds`, Excel (both `.xls` / `.xlsx`), `.json`, `.dbf` and `.dbc`. This package contains a few example datasets in different formats for you to load and test the function with. You can see all examples by passing the `d4h_example` function with no arguments: ```{r see_example_files} d4h_example() ``` The example below shows how to **load any file** with `d4h_load()`. The function will automatically detect the file format and load it accordingly. ```{r load_single} csv_path <- d4h_example("example_dataset.csv") head(d4h_load(csv_path)) ``` It is also possible to **load multiple files** at once, as long as the column names are consistent across files. The function will automatically bind the rows together into one data frame. ```{r load_multiple} rds2023_path <- d4h_example("example_dataset_2023.rds") rds2024_path <- d4h_example("example_dataset_2024.rds") # to check that both files were loaded correctly, # we will check the number of rows of each file nrow(d4h_load(rds2023_path)) nrow(d4h_load(rds2024_path)) # the number of rows corresponds to the sum of the rows in both files: nrow(d4h_load(c(rds2023_path, rds2024_path))) ``` In the case of **loading Excel files with multiple sheets**, you can specify the sheet name to load. If not specified, the function will load the first sheet by default. ```{r load_sheet} xlsx_path <- d4h_example("example_dataset.xlsx") d4h_load(xlsx_path) # as you can see all dates in DATE_INI, are from 2023 ``` You will see that the function automatically alerts you that there are multiple sheets in the Excel file and that it is loading the first sheet by default. If you want to load a specific sheet, you can specify the sheet name as follows: ```{r load_specific_sheet} d4h_load(xlsx_path, sheet = "year2024") # now, all dates in DATE_INI are from 2024 ``` `d4h_load()` checks whether the file contains a header or not. However you can force the function to treat the first row as a header or as data by using the `header` parameter. ```{r header} head(d4h_load(rds2023_path)) # the function automatically detects the header # forcing the first row as header head(d4h_load(rds2023_path, header = TRUE)) # forcing the first row to be included in the data head(d4h_load(rds2023_path, header = FALSE)) ``` For more information about `d4h_load()` and its parameters, see `?d4h_load`. --- ## 2. Cleaning data with `d4h_clean()` Here we show a selection of use cases. `d4h_clean()` handles the most common pre-processing tasks in one call, applying each step in the order listed below. This means later steps can reference outputs from earlier ones — for example, a renamed column can already be referred to by its new name within the same call. **Column selection** Depending on the number of columns in your data and the analysis you plan to do, you can use one of two parameters: * `cols_to_remove`: only specifying columns which you want to eliminate from your data. This is useful when you want to keep most of your columns and only want to remove a few of them. * `cols_to_include`: only specifying columns which you want to keep in your data. This is useful when you have many columns and you only want to keep a few of them. Currently, the dataset contains the following columns: ```{r colnames_1} colnames(example_dataset) ``` In our example, we want to remove the `CODE_GEO`, `LOCALCOD`, and `NOTES` columns, which are not relevant for our analysis. ```{r cols_to_remove} dengue_clean <- d4h_clean(data = example_dataset, cols_to_remove = c("CODE_GEO", "LOCALCOD", "NOTES")) ``` If you check the column names of dengue_clean, you will see that the columns have been removed: ```{r colnames_2} colnames(dengue_clean) ``` **Column renaming** You can either **apply a rule to all column names**, the options are: * lower: all to lowercase * upper: all to uppercase * no_spaces: replace all spaces by underscores If you want to change all column names to lowercase, you can use the following code: ```{r rename_columns_1} dengue_clean <- d4h_clean(data = dengue_clean, rename_columns = "lower") ``` Now, the column names of dengue_clean are: ```{r colnames_3} colnames(dengue_clean) ``` If you want to **rename column names to new names**, you can use the following code: ```{r rename_columns_2} dengue_clean <- d4h_clean(data = dengue_clean, rename_columns = c(name_geo = "region", date_brthd = "birthday_date", date_ini = "onset_date")) ``` Now the column names are: ```{r colnames_4} colnames(dengue_clean) ``` **Missing value filtering** Now we have to start to think about what variables are essential for our analysis. For example, if we want to do a time series analysis of dengue cases by region, we need to have a known onset date and a known region for each case. You can check for missing values in each column using: ```{r } colSums(is.na(dengue_clean)) ``` As you can see *onset_date* is complete but `region` has `r sum(is.na(dengue_clean$region))` missing values. If we want to drop these rows with missing values in these key variables, we can use the `remove_rows_missing` parameter in `d4h_clean()`. The following will drop any row that has a missing value in either `onset_date` or `region`. ```{r remove_rows_missing} dengue_clean <- d4h_clean(data = dengue_clean, remove_rows_missing = c("onset_date", "region")) ``` The function always gives an indication of how many rows were removed. We can double-check this by running: ```{r } nrow(dengue_clean) ``` **Sparse-column removal** Some columns may have a high proportion of missing values, which may reduce their usefulness for analysis. You can find the percentage of missing values with the following code: ```{r NA_percentage} colMeans(is.na(dengue_clean)) * 100 ``` In our example, the `serotype` column has `r mean(is.na(dengue_clean$serotype))*100`% missing values. You can choose to drop any column that has more than a certain percentage of missing values using the `threshold_remove` parameter. For example, the following will drop any column that has more than 50% missing values. ```{r threshold_remove} dengue_clean <- d4h_clean(data = dengue_clean, threshold_remove = 50) ``` As usual, the message printed in the console directly lets you know which columns were removed. **Category-recoding** Sometimes categorical variables are coded using abbreviations or numeric codes that are not human-readable (for example, ‘M’ instead of ‘Male’). You can use the `rename_categories` parameter to recode these values to more meaningful labels. For example, the following will recode the `sex` variable from "M" and "F" to "Male" and "Female", the `dengue_type` variable from "1", "2", "3" to "Dengue without warning signs", "Dengue with warning signs", and "Severe dengue", and the `hospitalised` variable from "S" and "N" to "Yes" and "No". ```{r rename_categories} dengue_clean <- d4h_clean(data = dengue_clean, rename_categories = list( sexo = c("M" = "Male", "F" = "Female"), dengue_type = c("1" = "Dengue without warning signs", "2" = "Dengue with warning signs", "3" = "Severe dengue"), hospitalised = c("S" = "Yes", "N" = "No") )) ``` **Date/time derivation** In epidemiology, time aggregation is essential because disease trends are typically analysed by epidemiological week, month, or year rather than by individual dates. In this package, we currently consider three timescales (epiweek, month, year) and two formats to represent time: date format (e.g., `2023-01-01`) and number format (e.g., `2023-01`). The same date can be represented in different formats. In the table below you can see the different representations of the same date (`2026-02-19`) in different formats and timescales: | | date | number | |---------|------------|---------| | epiweek | 15.02.2026 | 2026-07 | | month | 01.02.2026 | 2 | | year | 01.01.2026 | 2026 | *Note: The epiweek number is always given in combination with a year since the epidemiological weeks sometimes can have overlap, e.g. the first of January of 2026, corresponds to the epiweek 53 of the year 2025. If you need more information on epiweeks please check ISO 8601.* Please note that the start date of the week is by default set to Sunday as a European standard. You can change the start date by adding the parameter `week_start`. There are several functions to transform date columns from one to another: * `date_to_weekdate`: converts a date column to the date of the first day of the epiweek * `date_to_weeknumber`: converts a date column to the epiweek number (e.g., 2026-07) * `date_to_monthdate`: converts a date column to date of the first day of the month * `date_to_monthnumber`: converts a date column to the month number (e.g., 2 for February) * `date_to_yearnumber`: converts a date column to the year number (e.g., 2026) ```{r date_to_week} dengue_clean <- d4h_clean(data = dengue_clean, date_to_weekdate = "onset_date", date_to_weeknumber = "onset_date", date_to_monthdate = "onset_date", date_to_monthnumber = "onset_date", date_to_yearnumber = "onset_date" ) ``` You can see the results for one case here: ```{r} dengue_clean[1, ] ``` **Clean in one step** It is also possible to execute `d4h_clean()` only once with all the parameters at once, as in the example below: ```{r clean_all} dengue_clean <- d4h_clean( data = example_dataset, cols_to_remove = c("CODE_GEO", "LOCALCOD", "NOTES"), rename_columns = c( DATE_BRTHD = "birthdate", DATE_INI = "onset_date", NAME_GEO = "region", SEXO = "sex", DENGUE_TYPE = "dengue_type", HOSPITALISED = "hospitalised", RESULT_LABORATORY = "lab_result", SEROTYPE = "serotype" ), remove_rows_missing = c("onset_date", "region"), threshold_remove = 50, rename_categories = list( sex = c(M = "Male", F = "Female"), dengue_type = c("1" = "Dengue without warning signs", "2" = "Dengue with warning signs", "3" = "Severe dengue"), hospitalised = c(S = "Yes", N = "No")), date_to_weekdate = "onset_date", date_to_weeknumber = "onset_date", date_to_monthdate = "onset_date", date_to_monthnumber = "onset_date", date_to_yearnumber = "onset_date" ) # the resulting dataframe head(dengue_clean) ``` After cleaning, every row has a known onset date and sex, column names follow a consistent lower-case convention, and several derived time columns are ready for aggregation. For more information about `d4h_clean()` and its parameters, see `?d4h_clean`. --- ## 3. Filtering data with `d4h_filter()` `d4h_filter()` can be used to filter the dataset by any column you choose. Depending on the column type, different filter conditions are available: | Column type | Conditions | |-------------|---------------------------------------------| | numeric | `over`, `under`, `between`, `equal` | | Date | `after`, `before`, `during` | | character | `include`, `exclude`, `match`, `starts`, `ends` | To use `d4h_filter`, pass the column name you want to filter by as an argument, and the value of each argument is a list which contains the filter criteria. ```{r filter} dengue_filtered <- d4h_filter( data = dengue_clean, lab_result = list(include = "Positive"), onset_date = list(during = as.Date(c("2023-01-01", "2023-12-31"))), region = list(exclude = "Arendelle") ) nrow(dengue_filtered) ``` For more information about `d4h_filter()` and its parameters, see `?d4h_filter`. --- ## 4. Aggregating data with `d4h_aggregate()` `d4h_aggregate()` aggregates by counting the number of cases within defined groups (for example, the number of dengue cases per week and per region). This step transforms individual-level data into summary statistics suitable for reporting, visualisation, or modelling. For example, instead of analysing 500 individual patient records, you might analyse 52 weekly case counts. It also supports additional stratification variables via `add_col`, and can fill in zero-count cells for missing time-space combinations with `all_times` and `all_spaces`. **Simple spatio-temporal aggregation** ```{r agg-weekly} weekly_by_region <- d4h_aggregate( data = dengue_filtered, time_col = "onset_date_weekdate", space_col = "region" ) head(weekly_by_region) ``` **Monthly counts stratified by dengue severity** This is the typical input format for an epidemic curve or a time-series model. ```{r agg-monthly} monthly_by_type <- d4h_aggregate( data = dengue_filtered, time_col = "onset_date_monthdate", space_col = "region", add_col = "dengue_type" ) head(monthly_by_type, 12) ``` **Ensuring a complete time series** In surveillance data, weeks with zero reported cases are often not explicitly recorded. This can lead to misleading analyses if missing weeks are interpreted as missing data rather than true zero counts. Pass a full date sequence via `all_times` so that the aggregated output contains explicit zero rows rather than silently missing weeks. ```{r agg-complete} all_weeks <- seq( from = as.Date("2023-01-01"), to = as.Date("2023-12-31"), by = "week" ) weekly_complete <- d4h_aggregate( data = dengue_filtered, time_col = "onset_date_weekdate", space_col = "region", all_times = all_weeks, all_spaces = c("Lima", "Cusco", "Loreto") ) # confirm zeros are present sum(weekly_complete$cases == 0) ``` For more information about d4h_aggregate() and its parameters, see `?d4h_aggregate`. --- ## 5. Saving results with `d4h_save()` `d4h_save()` writes a `data.frame` to disk. The parameters are: * data: the `data.frame` to save * name: the name you want the file to have If you omit `name`, the file name will automatically include a timestamp (current date and time). * format: to indicate in what format the data should be saved. Supported formats are `"csv"`, `"csv2"`, `"rds"`, `"xls"`, `"dbf"`, and `"json"`. ```{r save, eval = FALSE} # Save the weekly aggregation as a CSV for sharing d4h_save( data = weekly_complete, name = "dengue_weekly_2023" ) # Save the cleaned individual-level data as RDS for fast reloading in R d4h_save( data = dengue_filtered, name = "dengue_analysis_clean", format = "rds" ) # Save for teams that use Excel d4h_save( data = monthly_by_type, name = "dengue_monthly_by_type", format = "xls" ) ``` For more information about d4h_save() and its parameters, see `?d4h_save`. --- ## Putting it all together Together, these five functions form a concise and readable data-processing pipeline. Using the base R pipe (`|>`, available since R 4.1) the full workflow above can be written as: ```{r pipeline, eval = FALSE} d4h_load(csv_path) |> d4h_clean( cols_to_remove = c("CODE_GEO", "LOCALCOD", "NOTES"), rename_columns = c( DATE_BRTHD = "birthdate", DATE_INI = "onset_date", NAME_GEO = "region", SEXO = "sex", DENGUE_TYPE = "dengue_type", HOSPITALISED = "hospitalised", RESULT_LABORATORY = "lab_result", SEROTYPE = "serotype" ), remove_rows_missing = c("onset_date", "region"), threshold_remove = 50, rename_categories = list( sex = c(M = "Male", F = "Female"), dengue_type = c("1" = "Dengue without warning signs", "2" = "Dengue with warning signs", "3" = "Severe dengue"), hospitalised = c(S = "Yes", N = "No")), date_to_weekdate = "onset_date", date_to_weeknumber = "onset_date", date_to_monthdate = "onset_date", date_to_monthnumber = "onset_date", date_to_yearnumber = "onset_date" ) |> d4h_filter( lab_result = list(include = "Positive"), onset_date = list(during = as.Date(c("2023-01-01", "2023-12-31"))), region = list(exclude = "Arendelle") ) |> d4h_aggregate( time_col = "onset_date_weekdate", space_col = "region", all_times = all_weeks, all_spaces = c("Lima", "Cusco", "Loreto") ) |> d4h_save(name = "dengue_weekly_confirmed", format = "csv") ``` The workflow is concise and readable, making it easy to follow the process from data loading to final output. Each step is clearly defined and can be easily modified or extended as needed. Once you have executed the full workflow, you will have a cleaned, filtered, and aggregated dataset that is ready for analysis or reporting. We recommend the following next steps: * Climate data processing via [clim4health](https://bsc-es.github.io/GHRtools/docs/clim4health/clim4health.html) * Data exploration via [GHRexplore](https://cran.r-project.org/package=GHRexplore) * Modelling via [GHRmodel](https://cran.r-project.org/package=GHRmodel) * Predicting via GHRpredict (coming soon!) The **data4health** package is designed to be flexible and adaptable to different types of health data and analysis needs. We encourage you to explore the documentation and experiment with the functions to see how they can best serve your specific use case. If you have any needs that are not currently met by the package, please feel free to reach out to us with suggestions for new features or improvements. We are committed to making this toolkit as useful and user-friendly as possible for the health data community.