Modules in {cleanepi}
In this section, we provide a detailed description of the way that
every module is built.
1. Standardization of column names
This module is designed to standardize the style and format of column
names within the target dataset, offering users the flexibility to
specify a subset of:
focal columns to preserve in their original format, and
columns to be renamed i.e. given a new name chosen by the
user.
Main function:
standardize_column_names()
Input:
- A
data.frame or linelist object.
- Optionally, a
vector of focal column names and a vector
of column names to be renamed in the form of
new_name = "old_name". If not provided, all columns will
undergo standardization.
Output:
- The input object with standardized column names.
Report:
- A two-column table displaying the initial and current column names
for each updated column in the original dataset.
Mode:
By incorporating the standardize_column_names()
function, {cleanepi} streamlines the process of ensuring consistency and
clarity in column naming conventions, thereby enhancing the overall
organization and readability of the dataset.
2. Removal of empty rows and columns and constant
columns
This module aims at eliminating irrelevant and redundant rows and
columns, including empty rows and columns as well as constant
columns.
- Main function:
remove_constants()
- Input: Accepts a
data.frame or
linelist object, along with:
- A cut-off that determines the percent of missing values beyond which
a row or column should be deleted.
- Output: Returns the input object after applying the
specified operations.
- Report:
- A list with one or two or three vector(s) showcasing detected empty
rows, empty columns and constant columns.
- Mode:
3. Detection and removal of duplicates
This module is designed to identify and eliminate duplicated
rows.
- Main function:
find_duplicates(), remove_duplicates()
- Input: Accepts a
data.frame or
linelist object, along with optional parameters:
- Vector of target columns (default is to consider all columns;
possible to use
linelist_tags to consider tagged variables
only when the input is a linelist object).
- Output: Returns the input object after applying the
specified operations.
- Report:
- A list with one or two table(s) showcasing found duplicates and
removed duplicates (if
remove = TRUE).
- A table detailing the removed duplicates.
- Mode:
Through the remove_duplicates() function, users can
streamline their dataset by eliminating redundant rows, thus enhancing
data integrity and analysis efficiency.
4. Replacement of missing values with
NA
This module aims to standardize and unify the representation of
missing values within the dataset.
- Main function:
replace_missing_values()
- Input: Accepts a
data.frame or
linelist object, along with:
- A
vector of column names (if not provided, the
operation is performed across all columns)
- A string or a vector of strings representing the current missing
values (default value is cleanepi::common_na_strings)
- Output: Returns the input object with all missing
values represented as
NA.
- Report: Generates a three-column table featuring
index, column, and value for each missing item in the dataset.
- Mode:
By utilizing the replace_missing_char() function, users
can ensure consistency in handling missing values across their dataset,
facilitating accurate analysis and interpretation of the data.
5. Standardization of date values
This module is dedicated to convert date values in character columns
into Date value in ISO8601 format, and
ensuring that all dates fall within the given timeframe.
- Main function:
standardize_dates()
- Input: Accepts a
data.frame or
linelist object, along with:
- A
vector of targeted date columns (automatically
determined if not provided)
- Tolerance threshold that defines the % of missing (out of range
values converted to
NA) values to be allowed in a converted
column. When % missing values exceeds or is equal to it, the original
values are returned (default value is 40%)
- format (default value is NULL)
- timeframe (default value is null)
- Output: Returns the input object with standardized
date values in the format of %Y-%m-%d.
- Report:
- A two-column table listing the columns where date values were
standardized.
- A three-column table featuring index, column name, and values that
fall outside the specified timeframe.
- A data frame featuring date values that can comply with more than
one specified format. Users can update the standardized date values with
the correct when it’s appropriated.
- Mode:
By employing the standardize_dates() function, users can
ensure uniformity and coherence in date formats across their dataset,
while also validating the temporal integrity of the data within the
defined timeframe.
6. Standardization of subject IDs
This module is tailored to verify whether the values in the column
uniquely identifying subjects adhere to a consistent format. It also
offers a functionality that allow users to correct the inconsistent
subject ids.
- Main function:
check_subject_ids(), correct_subject_ids()
- Input: Accepts a
data.frame or
linelist object, along with:
- The name of the ID column
- Strings for prefix, suffix, and numerical range within the IDs
- A logical that determines whether to delete the wrong ids or
not.
- Output: Returns the input object with standardized
subject IDs.
- Report: Generates a two-column table featuring
index and value of each subject ID that deviates from the expected
format.
- Mode:
By utilizing the functions in this module, users can ensure
uniformity in the format of subject ids, facilitating accurate tracking
and analysis of individual subjects within the dataset.
7. Dictionary based substitution
This module facilitates dictionary-based substitution, which involves
replacing existing values with predefined ones. It replaces entries in a
specific columns to certain values, such as substituting 1 with “male”
and 2 with “female” in a gender column. It also interoperates seamlessly
with the get_meta_data() function from {readepi} R
package.
- Main function:
clean_using_dictionary()
- Input: Accepts a
data.frame or
linelist object, along with a data dictionary featuring the
following column names: options, values, and
order.
- Output: Returns the input object where the
specified options are replaced by their corresponding values.
- Report: Generates a three-column table with index,
column, and value for each unexpected value encountered in a targeted
column.
- Mode:
By leveraging the clean_using_dictionary() function,
users can streamline and standardize the values within specific columns
based on predefined mappings, enhancing consistency and accuracy in the
dataset.
Note that the clean_using_dictionary() function will
return a warning when it detects unexpected values in the target columns
from the data dictionary. The unexpected values can be added to the data
dictionary using the add_to_dictionary() function.
8. Conversion of values when necessary
This module is designed to convert numbers written in letters to
numerical values, ensuring interoperability with the
{numberize} package.
- Main function:
convert_to_numeric()
- Input: Accepts a
data.frame or
linelist object, along with:
- A vector of column names to be converted into numeric, and
- A string that denotes the language used in the text. The current
version supports
English, French and Spanish.
- Output: Returns the input object with values in the
target columns converted into numeric format.
- Report: Generates a three-column table with index,
column, and value for each unrecognized value in the dataset (strings
that could not be converted into numeric).
- Mode:
By employing the convert_to_numeric() function, users
can seamlessly transform numeric representations written in letters into
numerical values, ensuring compatibility with the {numberize} package
and promoting accuracy in numerical analysis.
Note that convert_to_numeric() will issue a warning for
unexpected values and return them in the report.
9. Verification of the sequence of date-events
This module provides functions to verify whether the sequence of date
events aligns with expectations. For instance, it can flag rows where
the date of admission to the hospital precedes the individual’s date of
birth.
- Main function:
check_date_sequence()
- Input: Accepts a
data.frame or
linelist object, along with:
- A vector of date column names to be considered
- Output: Returns the input object with incorrect
rows removed if specified.
- Report: Generates a table listing the incorrect
rows from the specified columns.
- Mode:
By using the check_date_sequence() function, users can
systematically validate and ensure the coherence of date sequences
within their dataset, promoting accuracy and reliability in subsequent
analyses.
10. Transformation of selected columns
This module is dedicated to performing various specialized operations
related to epidemiological data analytics, and it currently includes the
following functions:
- Main function:
timespan()
- Input: Accepts a
data.frame or
linelist object, along with:
- The name of the column of interest
- The reference date value or column
- The time and remainder units (possible values are days, weeks,
months, or years)
- The names of the newly generated columns
- Output: Returns the input object with clean data
and one or two additional columns with values in the specified time
unit.
- Report: None.
- Mode:
By leveraging the timespan() function, users can
efficiently compute and integrate time span information into their
epidemiological dataset based on user-defined parameters, enhancing the
analytics capabilities of the dataset.
