RISQ is a package for the estimation of Representativity Indicators for Survey Quality. That is, it provides metrics that can be used to evaluate survey quality.
The main functions of the RISQ package are:
risq(): Create a RISQ object for a data set.rr(): Estimate response rate for a RISQ object.ri(), ri_by_var(),
ri_by_cat(): Estimate R-indicator for a RISQ object.cv(), cv_by_var(),
cv_by_cat(): Estimate coefficient of variation for a RISQ
object.bootstrap(): Create a bootstrap object for a RISQ
object and indicator function.mean(), var(), quantile() and
- for bootstrap objects.The RISQ package provides the dataset hlc for testing
and tutorial purposes. The hlc dataset is a public subset
of data from a survey on Household Living Conditions conducted by
Statistics Netherlands.
head(hlc)
#> age gender marital_status household job house_value
#> 1 0-17 years Female Not married Couple with children No 5
#> 2 0-17 years Female Not married Couple with children No 6
#> 3 0-17 years Male Not married Couple with children No 6
#> 4 0-17 years Male Not married Single parent No 4
#> 5 0-17 years Male Not married Couple with children No 6
#> 6 0-17 years Female Not married Couple with children No 5
#> urbanisation response_3 response_2 response_1
#> 1 Strong FALSE FALSE FALSE
#> 2 Very strong FALSE FALSE FALSE
#> 3 Little TRUE FALSE FALSE
#> 4 Very strong FALSE FALSE FALSE
#> 5 Not FALSE FALSE FALSE
#> 6 Strong FALSE FALSE FALSEIn this introduction, we use a subset of the hlc data to
limit the computation time of the examples.
Indicator functions take a RISQ object as first argument. This way, a RISQ object can be created once, and used as input for all indicator functions.
A RISQ object includes data, a model formula for estimating response propensities, the regression type, and sampling weights and strata.
The data must hold a record for each unit in the sample, with information relevant to the model. There should be at least one target variable that describes whether the unit did or did not respond to the survey. Note that it is assumed that administrative non-response and ineligible units have been omitted prior to evaluation. RISQ does not distinguish between different types of non-response, and an adjustment for non-observed ineligibility is not possible. RISQ supports only RR1, according to AAPOR guidelines.
The variables used in the response model formula must all be present in the data.
The regression type is binomial by default, but can also
be specified as gaussian.
It is assumed that the sampling design has a fixed sample size, and population units are selected without replacement. The type of sampling design is derived from the sampling weights and strata. If both weights and strata are omitted, it is assumed that the data set is a full enumeration, i.e. all population units have been selected. If there is a single stratum with constant weights, a simple random sample (SI) design is assumed. If there are multiple strata with constant weights within each stratum, a stratified simple random sample (STSI) design is assumed. If sampling weights are not constant within each stratum, a probability-proportional-to-size (PPS) design is assumed.
The example below creates a RISQ object with four predictors included as main effects, based on a simple random sample with sampling weights equal to 250.
model <- formula(~ age + gender + house_value + household)
weights <- rep(250, nrow(data))
risq_data <- risq(predictor = model, family = "binomial", data = data, weights = weights)
risq_data
#> model:
#> predictor: ~age + gender + house_value + household
#> family: binomial
#> data:
#> samples: 887
#> variables: 10
#> design:
#> weights:
#> values: 887
#> unique: 1
#> strata:
#> values: 887
#> levels: 1
#> type: SIR-indicators and coefficients of variation are the main indicators provided by the RISQ package. The response rate plays a key part in the estimation of these indicators.
The functions for indicator estimation take a RISQ object and a target variable as input. The target variable must be a logical variable in the RISQ object data, describing whether the sample unit responded or not.
The response rate estimates the proportion of the sample that responded, based on the response model.
The R-indicator measures the standard deviation in estimated response propensities for the specified set of variables. It it based on the distance to constant response propensities from a so-called weak representative response perspective. R-indicators are transformed to a scale from 0 to 1, such that 0 is the worst and 1 is the best case.
The coefficient of variation is an additional measure that equals the standard deviation of estimated response propensities over the response rate. It approximates the non-response bias for means and sums based on the response. coefficients of variation are non-negative values, with 0 being the best case.
Partial R-indicators and coefficients of variation at the variable
level can be used to determine the contribution of single variables.
These partial indicators can be estimated using the
ri_by_var() and cv_by_var() functions. The
variables for which to estimate indicator values, must be provided as
input to the partial indicator function. Any categorical variable that
is present in the data can be used, including variables that are not
present in the regression model.
Partial indicators come in two types, unconditional and
conditional. Unconditional indicators correspond to the
total contribution of a variable, whereas conditional indicators show
the unique contributions. At variable level, partial R-indicators have
values between 0 and 0.5 and coefficients of variation are non-negative.
In both cases, the ideal value is 0.
Note that for variables not present in the regression model, only unconditional indicator values make sense. Unconditional indicators are always 0 for such variables.
As an example, we estimate the unconditional and conditional partial
R-indicators for the variables age and
marital_status.
ri_by_var(risq_data, target = "response_3", variables = c("age", "marital_status"), type = "unconditional")
#> variable value se
#> 1 age 0.04801174 0.01438862
#> 2 marital_status 0.02016297 0.01699730
ri_by_var(risq_data, target = "response_3", variables = c("age", "marital_status"), type = "conditional")
#> variable value se
#> 1 age 0.04492495 0.01438862
#> 2 marital_status 0.00000000 0.00000000Partial R-indicators and coefficients of variation at the category
level can be used to zoom in to the contribution of single categories of
variables. These partial indicators can be estimated using the
ri_by_cat() and cv_by_cat() functions. Again,
the variables for which to estimate indicator values must be provided as
input to the partial indicator function. Any categorical variable that
is present in the data can be used, including variables that are not
present in the regression model. Estimates are provided for all
categories of each of the specified variables.
The partial indicators can be categorized as
unconditional for total contribution, or
conditional for unique contribution. At variable level,
conditional partial R-indicators have values between 0 and 0.5 and
conditional coefficients of variation are non-negative. Unconditional
partial indicators are similar, but include a sign. A positive
unconditional partial indicator suggests overrepresentation, while a
negative unconditional partial indicator suggests underrepresentation.
In all cases, the ideal value is 0.
Here too, only unconditional indicator values make sense for variables not present in the regression model.
As an example, we estimate the unconditional and conditional partial
R-indicators for the categories of the variables gender and
job.
ri_by_cat(risq_data, target = "response_3", variables = c("gender", "job"), type = "unconditional")
#> variable category value se
#> 1 gender Male -0.010083527 0.01220574
#> 2 gender Female 0.010255515 0.01241393
#> 3 job No 0.006913243 0.01254946
#> 4 job Yes -0.006660653 0.01209094
ri_by_cat(risq_data, target = "response_3", variables = c("gender", "job"), type = "conditional")
#> variable category value se
#> 1 gender Male 0.005475043 0.01189432
#> 2 gender Female 0.005791698 0.01257315
#> 3 job No 0.000000000 0.00000000
#> 4 job Yes 0.000000000 0.00000000For some values of interest related to the representativity indicators for survey quality, no general analytical approximation is known. Important examples are indicator quantiles and the standard errors for the difference between different target variables of the same survey. To allow the estimation of these values, RISQ implements bootstrapping of indicators using bootstrap objects.
Below, bootstrap objects are created for the R-indicator of two
different response variables. Note that the iteration count is low to
limit the computing time of the example. In real applications, an
iteration count of 500 is often a good starting point.
bs_ri_resp1 <- bootstrap(risq_data, fun = ri, seed = 0, target = "response_1", iterations = 10)
bs_ri_resp3 <- bootstrap(risq_data, fun = ri, seed = 0, target = "response_3", iterations = 10)The statistical functions mean(), var() and
quantile() have been generalized to work with bootstrap
objects. Further, the difference between two bootstrap objects can be
calculated using the - operator. This operator should only
be used on bootstrap objects created using the same RISQ object, with
the same fun, seed and iterations
arguments.
The mean and standard error (and therefore the variance) of an indicator can be estimated directly using the appropriate indicator function. Quantiles, however, can only be estimated using bootstrapping. The below example shows how to estimate the 2.5% quantiles of an indicator, using its bootstrap object.
RISQ allows the estimation of differences between indicators for different target variables of the same data. For the difference of indicator values, without standard errors, individual indicator values can simply be subtracted.
ri_resp1 <- ri(risq_data, target = "response_1", include_se = FALSE)
ri_resp3 <- ri(risq_data, target = "response_3", include_se = FALSE)
ri_resp3$value - ri_resp1$value
#> [1] -0.1405905The same value can be estimated using the mean of the difference of two bootstrap objects.
The variance, standard error, and quantiles of indicator differences between target variables, can only be estimated using bootstrapping.