Data Needs

To calculate response rates, it’s not enough to look at just the data from individuals who ultimately responded to the survey; we need a dataset that includes every individual from whom a response was sought. For example, if a school sent out an email survey to parents, then to calculate response rates we would need a list of all the parents to whom an email was sent, regardless of whether that parent ultimately responded. For example, we would need a data file with a response status variable for each parent, as in the example table below:

With such a dataset, we need to classify each individual’s response status into one of four categories:

• Ineligible: This individual was asked to complete the survey, but it was discovered that they were not eligible. For example, if a person was invited to complete a school parent survey and that person replied that they are not in fact a parent, that person would be classified as ineligible.

• Eligible Respondent: This individual completed the survey and was in fact eligible to do so.

• Eligible Nonrespondent: This individual did not complete the survey, but it is known that they were eligible to do so. For example, a parent may respond to a school’s emails for purposes such as communicating with teachers, but they may not respond to a survey invitation sent to their email. In this example, it is thus known that the person is a nonrespondent and that they were in fact eligible for the survey.

• Unknown Eligibility: It is unknown whether this individual was eligible to complete the survey. For example, suppose a mail survey was sent to a random sample of addresses in a school district, where the survey was meant to include only parents of school-aged children. If a given address never replies to the survey and it is thus unknown whether any parents live at that address, then the case would be classified as having unknown eligibility.

Specifying a Response Rate Analysis

The function calculate_response_rates() can be used to calculate the response rate for a survey once all of the records in the data have been grouped into the four categories. The argument status identifies the variable in the data used to indicate response status, and the argument status_codes allows the user to specify how the categories of that variable should be interpreted. The result is a data frame, with the response rate given in the column RR3_Unweighted and the underlying counts given in the columns n, n_ER, n_EN, etc.

# Load example data
data('involvement_survey_srs', package = "nrba")

# Calculate overall response rates for the survey
calculate_response_rates(
  data = involvement_survey_srs,
  status = "RESPONSE_STATUS",
  status_codes = c(
    'ER' = 'Respondent',
    'EN' = 'Nonrespondent',
    'IE' = 'Ineligible',
    'UE' = 'Unknown'
  ),
  rr_formula = 'RR1'
)
#>   RR1_Unweighted    n n_ER n_EN n_IE n_UE
#> 1      0.6223203 5000 2961 1563  242  234

If the survey uses weights to account for unequal probabilities of selection, then the name of a weight variable can be supplied to the weights argument. The output variables Nhat, Nhat_ER, etc. provide weighted versions of the variables n, n_ER, etc.

# Load example data
data('involvement_survey_str2s', package = "nrba")

# Calculate overall response rates for the survey
calculate_response_rates(
  data = involvement_survey_str2s,
  weights = "BASE_WEIGHT",
  status = "RESPONSE_STATUS",
  status_codes = c(
    'ER' = 'Respondent',
    'EN' = 'Nonrespondent',
    'IE' = 'Ineligible',
    'UE' = 'Unknown'
  ),
  rr_formula = 'RR1'
)
#>   RR1_Unweighted RR1_Weighted    n    Nhat n_ER Nhat_ER n_EN Nhat_EN n_IE
#> 1      0.5747126    0.5662812 1000 20268.6  550   11007  363 7672.66   43
#>   Nhat_IE n_UE Nhat_UE
#> 1  831.26   44  757.68

Calculate response rates by group

To calculate response rates separately by groups, we can first group the input data using the group_by() function from the popular ‘dplyr’ package.

library(dplyr)

involvement_survey_srs |>
  group_by(STUDENT_RACE) |>
  calculate_response_rates(
    status = "RESPONSE_STATUS",
    status_codes = c(
      'ER' = 'Respondent',
      'EN' = 'Nonrespondent',
      'IE' = 'Ineligible',
      'UE' = 'Unknown'
    ),
    rr_formula = 'RR1'
  )
#> # A tibble: 7 × 7
#>   STUDENT_RACE                             RR1_U…¹     n  n_ER  n_EN  n_IE  n_UE
#>   <chr>                                      <dbl> <int> <int> <int> <int> <int>
#> 1 AM7 (American Indian or Alaska Native)     0.744    47    32    10     4     1
#> 2 AS7 (Asian)                                0.674    44    29    12     1     2
#> 3 BL7 (Black or African American)            0.688   694   457   169    30    38
#> 4 HI7 (Hispanic or Latino Ethnicity)         0.350   948   314   542    50    42
#> 5 MU7 (Two or More Races)                    0.662   138    88    39     5     6
#> 6 PI7 (Native Hawaiian or Other Pacific I…   0.677    35    21    10     4     0
#> 7 WH7 (White)                                0.686  3094  2020   781   148   145
#> # … with abbreviated variable name ¹​RR1_Unweighted

Formulas for calculating response rates

When every person invited to participate in a survey is known to be eligible for the survey, it is quite easy to calculate a response rate: simply count the number of respondents and divide this count by the total number of respondents and nonrespondents. Response rate calculations become more complicated when there are cases with unknown eligibility.

When there are cases with unknown eligibility, the common convention is to use one of the response rate formulas promulgated by the American Association for Public Opinion Research (AAPOR); see @theamericanassociationforpublicopinionresearchStandardDefinitionsFinal2016. The three most commonly-used formulas are referred to as “RR1”, “RR3”, and “RR5”. Response rates can be calculated using one or more formulas by supplying the formula names to the rr_formula argument of calculate_response_rates().

#>   RR1_Unweighted RR3_Unweighted RR5_Unweighted    n n_ER n_EN n_IE n_UE
#> 1      0.6223203      0.6238782      0.6545093 5000 2961 1563  242  234
#>     e_unwtd
#> 1 0.9492237

These formulas differ only in how many cases with unknown eligibility are estimated to in fact be eligible for the survey.

\[ \begin{aligned} RR1 &= ER / (ER + EN + UE) \\ RR3 &= ER / (ER + EN + (e \times UE)) \\ RR5 &= ER / (ER + EN) \\ &\text{where:} \\ ER &\text{ is the total number of eligible respondents} \\ EN &\text{ is the total number of eligible nonrespondents} \\ UE &\text{ is the total number of cases whose eligibility is unknown} \\ &\text{and} \\ e &\text{ is an *estimate* of the percent of unknown eligibility cases} \\ &\text{which are in fact eligible} \end{aligned} \] For the RR3 formula, it is necessary to produce an estimate of the share of unknown eligibility cases who are in fact eligible, denoted \(e\). One common estimation method is the “CASRO” method: among cases with known eligibility status, calculate the percent who are known to be eligible.

\[ \begin{aligned} \text{CASRO}&\text{ method:} \\ e &= 1 - \frac{IE}{IE + ER + EN} \\ &\text{where:} \\ IE &\text{ is the total number of sampled cases known to be ineligible} \\ \end{aligned} \]

When calculating response rates for population subgroups, one can either assume the eligibility rate \(e\) is constant across all subgroups or one can estimate the eligibility rate separately for each subgroup. When using the CASRO method to estimate the eligibility rate, the former approach is referred to as the “CASRO overall” method, while the latter approach is referred to as the “CASRO subgroup” method. Either option can be used by specifying either elig_method='CASRO-overall' or elig_method='CASRO-subgroup'.

involvement_survey_srs |>
  group_by(PARENT_HAS_EMAIL) |>
  calculate_response_rates(
    status = "RESPONSE_STATUS",
    status_codes = c(
      'ER' = 'Respondent',
      'EN' = 'Nonrespondent',
      'IE' = 'Ineligible',
      'UE' = 'Unknown'
    ),
    rr_formula = 'RR3',
    elig_method = "CASRO-subgroup"
  )
#> # A tibble: 2 × 8
#>   PARENT_HAS_EMAIL RR3_Unweighted     n  n_ER  n_EN  n_IE  n_UE e_unwtd
#>   <chr>                     <dbl> <int> <int> <int> <int> <int>   <dbl>
#> 1 Has Email                 0.626  4262  2533  1319   204   206   0.950
#> 2 No Email                  0.613   738   428   244    38    28   0.946

Alternatively, the user may specify a specific value of \(e\) to use for response rates or a variable in the data to use which specifies a value of \(e\) for different groups.

involvement_survey_srs %>%
  mutate(e_by_email = ifelse(PARENT_HAS_EMAIL == 'Has Email', 0.75, 0.25)) %>%
  group_by(PARENT_HAS_EMAIL) %>%
  calculate_response_rates(status = "RESPONSE_STATUS",
                           status_codes = c(
                             'ER' = 'Respondent',
                             'EN' = 'Nonrespondent',
                             'IE' = 'Ineligible',
                             'UE' = 'Unknown'
                           ),
                           rr_formula = "RR3",
                           elig_method = "specified",
                           e = "e_by_email")
#> # A tibble: 2 × 8
#>   PARENT_HAS_EMAIL RR3_Unweighted     n  n_ER  n_EN  n_IE  n_UE e_unwtd
#>   <chr>                     <dbl> <int> <int> <int> <int> <int>   <dbl>
#> 1 Has Email                 0.632  4262  2533  1319   204   206    0.75
#> 2 No Email                  0.630   738   428   244    38    28    0.25