Contrasts

In a previous vignette, we introduced the “marginal effect” as a partial derivative. Since derivatives are only properly defined for continuous variables, we cannot use them to interpret the effects of changes in categorical variables. For this, we turn to contrasts between Adjusted predictions. In the context of this package, a “Contrast” is defined as:

The difference between two adjusted predictions, calculated for meaningfully different regressor values (e.g., College graduates vs. Others).

The marginaleffects() function automatically calculates contrasts instead of derivatives for factor, logical, or character variables.

The comparisons() function gives users more powerful features to compute different contrasts. For example, it allows users to create custom contrasts like ratios instead of differences, and with different values of the predictors.

Contrasts: logical and factor variables

Consider a simple model with a logical and a factor variable:

library(marginaleffects)
library(magrittr)

tmp <- mtcars
tmp$am <- as.logical(tmp$am)
mod <- lm(mpg ~ am + factor(cyl), tmp)

The marginaleffects function automatically computes contrasts for each level of the categorical variables, relative to the baseline category (FALSE for logicals, and the reference level for factors), while holding all other values at their mode or mean:

mfx <- marginaleffects(mod)
summary(mfx)
#> Average marginal effects 
#>   Term     Contrast  Effect Std. Error z value   Pr(>|z|)     2.5 % 97.5 %
#> 1   am TRUE - FALSE   2.560      1.298   1.973    0.04851   0.01675  5.103
#> 2  cyl        6 - 4  -6.156      1.536  -4.009 6.1077e-05  -9.16608 -3.146
#> 3  cyl        8 - 4 -10.068      1.452  -6.933 4.1147e-12 -12.91359 -7.222
#> 
#> Model type:  lm 
#> Prediction type:  response

The summary printed above says that moving from the reference category 4 to the level 6 on the cyl factor variable is associated with a change of -6.156 in the adjusted prediction. Similarly, the contrast from FALSE to TRUE on the am variable is equal to 2.560.

We can obtain different contrasts by using the comparisons() function. For example:

comparisons(mod, contrast_factor = "sequential") %>% tidy()
#>       type term     contrast  estimate std.error statistic      p.value
#> 1 response   am TRUE - FALSE  2.559954  1.297579  1.972869 4.851044e-02
#> 2 response  cyl        6 - 4 -6.156118  1.535723 -4.008612 6.107658e-05
#> 3 response  cyl        8 - 6 -3.911442  1.470254 -2.660385 7.805144e-03
#>      conf.low conf.high
#> 1  0.01674586  5.103162
#> 2 -9.16607927 -3.146156
#> 3 -6.79308707 -1.029797
comparisons(mod, contrast_factor = "pairwise") %>% tidy()
#>       type term     contrast   estimate std.error statistic      p.value
#> 1 response   am TRUE - FALSE   2.559954  1.297579  1.972869 4.851044e-02
#> 2 response  cyl        6 - 4  -6.156118  1.535723 -4.008612 6.107658e-05
#> 3 response  cyl        8 - 4 -10.067560  1.452082 -6.933187 4.114709e-12
#> 4 response  cyl        8 - 6  -3.911442  1.470254 -2.660385 7.805144e-03
#>       conf.low conf.high
#> 1   0.01674586  5.103162
#> 2  -9.16607927 -3.146156
#> 3 -12.91358877 -7.221530
#> 4  -6.79308707 -1.029797
comparisons(mod, contrast_factor = "reference") %>% tidy()
#>       type term     contrast   estimate std.error statistic      p.value
#> 1 response   am TRUE - FALSE   2.559954  1.297579  1.972869 4.851044e-02
#> 2 response  cyl        6 - 4  -6.156118  1.535723 -4.008612 6.107658e-05
#> 3 response  cyl        8 - 4 -10.067560  1.452082 -6.933187 4.114709e-12
#>       conf.low conf.high
#> 1   0.01674586  5.103162
#> 2  -9.16607927 -3.146156
#> 3 -12.91358877 -7.221530

For comparison, this code produces the same results using the emmeans package:

library(emmeans)
emm <- emmeans(mod, specs = "cyl")
contrast(emm, method = "revpairwise")
#>  contrast    estimate   SE df t.ratio p.value
#>  cyl6 - cyl4    -6.16 1.54 28  -4.009  0.0012
#>  cyl8 - cyl4   -10.07 1.45 28  -6.933  <.0001
#>  cyl8 - cyl6    -3.91 1.47 28  -2.660  0.0331
#> 
#> Results are averaged over the levels of: am 
#> P value adjustment: tukey method for comparing a family of 3 estimates

emm <- emmeans(mod, specs = "am")
contrast(emm, method = "revpairwise")
#>  contrast     estimate  SE df t.ratio p.value
#>  TRUE - FALSE     2.56 1.3 28   1.973  0.0585
#> 
#> Results are averaged over the levels of: cyl

Note that these commands also work on for other types of models, such as GLMs, on different scales:

mod_logit <- glm(am ~ factor(gear), data = mtcars, family = binomial)

comparisons(mod_logit) %>% tidy()
#>       type term contrast  estimate    std.error    statistic      p.value
#> 1 response gear    4 - 3 0.6666667 1.360805e-01     4.899061 9.629569e-07
#> 2 response gear    5 - 3 1.0000000 1.071403e-05 93335.529606 0.000000e+00
#>    conf.low conf.high
#> 1 0.3999538 0.9333795
#> 2 0.9999790 1.0000210

comparisons(mod_logit, type = "link") %>% tidy()
#>   type term contrast estimate std.error   statistic   p.value   conf.low
#> 1 link gear    4 - 3 21.25922  4577.962 0.004643817 0.9962948  -8951.381
#> 2 link gear    5 - 3 41.13214  9155.924 0.004492407 0.9964156 -17904.148
#>   conf.high
#> 1   8993.90
#> 2  17986.41

Contrasts: numeric variables

We can also compute contrasts for differences in numeric variables. For example, we can see what happens to the adjusted predictions when we increment the hp variable by 1 unit (default) or by 5 units:

mod <- lm(mpg ~ hp, data = mtcars)

comparisons(mod) %>% tidy()
#>       type term    contrast    estimate std.error statistic      p.value
#> 1 response   hp (x + 1) - x -0.06822828 0.0101193 -6.742389 1.558043e-11
#>      conf.low   conf.high
#> 1 -0.08806175 -0.04839481

comparisons(mod, contrast_numeric = 5) %>% tidy()
#>       type term    contrast   estimate  std.error statistic      p.value
#> 1 response   hp (x + 5) - x -0.3411414 0.05059652 -6.742389 1.558043e-11
#>     conf.low conf.high
#> 1 -0.4403087 -0.241974

Compare adjusted predictions for a change in the regressor between two arbitrary values:

comparisons(mod, contrast_numeric = c(90, 110)) %>% tidy()
#>       type term contrast  estimate std.error statistic      p.value  conf.low
#> 1 response   hp 110 - 90 -1.364566 0.2023861 -6.742389 1.558043e-11 -1.761235
#>    conf.high
#> 1 -0.9678961

Compare adjusted predictions when the regressor changes across the interquartile range, across one or two standard deviations about its mean, or from across its full range:

comparisons(mod, contrast_numeric = "iqr") %>% tidy()
#>       type term contrast  estimate std.error statistic      p.value  conf.low
#> 1 response   hp  Q3 - Q1 -5.697061 0.8449619 -6.742389 1.558043e-11 -7.353156
#>   conf.high
#> 1 -4.040966

comparisons(mod, contrast_numeric = "sd") %>% tidy()
#>       type term                contrast  estimate std.error statistic
#> 1 response   hp (x + sd/2) - (x - sd/2) -4.677926 0.6938085 -6.742389
#>        p.value  conf.low conf.high
#> 1 1.558043e-11 -6.037766 -3.318087

comparisons(mod, contrast_numeric = "2sd") %>% tidy()
#>       type term            contrast  estimate std.error statistic      p.value
#> 1 response   hp (x + sd) - (x - sd) -9.355853  1.387617 -6.742389 1.558043e-11
#>    conf.low conf.high
#> 1 -12.07553 -6.636174

comparisons(mod, contrast_numeric = "minmax") %>% tidy()
#>       type term  contrast estimate std.error statistic      p.value  conf.low
#> 1 response   hp Max - Min -19.3086  2.863763 -6.742389 1.558043e-11 -24.92147
#>   conf.high
#> 1 -13.69573

Interactions between contrasts

In some contexts we would like to know what happens when two (or more) predictors change at the same time. In the marginaleffects package terminology, this is an “interaction between contrasts.”

For example, consider a model with two factor variables:

mod <- lm(mpg ~ am * factor(cyl), data = mtcars)

What happens if am increases by 1 unit and cyl changes from a baseline reference to another level?

cmp <- comparisons(mod, variables = c("cyl", "am"))
summary(cmp)
#> Average contrasts 
#>     cyl          am Effect Std. Error z value  Pr(>|z|)   2.5 % 97.5 %
#> 1 4 - 4 (x + 1) - x  5.175      2.053  2.5209 0.0117059   1.151  9.199
#> 2 6 - 4 (x + 1) - x -1.983      2.493 -0.7956 0.4262446  -6.869  2.902
#> 3 8 - 4 (x + 1) - x -7.048      2.731 -2.5805 0.0098656 -12.401 -1.695
#> 
#> Model type:  lm 
#> Prediction type:  response

When the variables argument is used and the model formula includes interactions, the “cross-contrasts” contrasts will automatically be displayed. You can also force comparisons() to do it by setting interactions=TRUE and using the variables argument to specify which variables should be manipulated simultaneously.

Contrast types: “Unit-Level”, “Average”, “At Mean”, “Between Marginal Means”

This section compares 4 quantities:

  1. Unit-Level Contrasts
  2. Average Contrast
  3. Contrast at the Mean
  4. Contrast Between Marginal Means

The ideas discussed in this section focus on contrasts, but they carry over directly to analogous types of marginal effects.

Unit-level contrasts

In models with interactions or non-linear components (e.g., link function), the value of a contrast or marginal effect can depend on the value of all the predictors in the model. As a result, contrasts and marginal effects are fundamentally unit-level quantities. The effect of a 1 unit increase in \(X\) can be different for Mary or John. Every row of a dataset has a different contrast and marginal effect.

The mtcars dataset has 32 rows, so the comparisons() function produces 32 contrast estimates:

Average contrasts

By default, the marginaleffects() and comparisons() functions compute marginal effects and contrasts for every row of the original dataset. These unit-level estimates can be unwieldy and hard to interpret. To help interpretation, the summary() function computes the “Average Marginal Effect” or “Average Contrast,” by taking the mean of all the unit-level estimates.

which is equivalent to:

We could also show the full distribution of contrasts across our dataset with a histogram:

This graph display the effect of a change of 1 unit in the mpg variable, for each individual in the observed data.

Contrasts at the mean

An alternative which used to be very common but has now fallen into a bit of disfavor is to compute “Contrasts at the mean.” The idea is to create a “synthetic” or “hypothetical” individual (row of the dataset) whose characteristics are completely average. Then, we compute and report the contrast for this specific hypothetical individual.

This can be achieved by setting newdata="mean" or to newdata=datagrid(), both of which fix variables to their means or modes:

Contrasts at the mean can differ substantially from average contrasts.

The advantage of this approach is that it is very cheap and fast computationally. The disadvantage is that the interpretation is somewhat ambiguous. Often times, there simply does not exist an individual who is perfectly average across all dimensions of the dataset. It is also not clear why the analyst should be particularly interested in the contrast for this one, synthetic, perfectly average individual.

Contrasts between marginal means

Yet another type of contrast is the “Contrast between marginal means.” This type of contrast is closely related to the “Contrast at the mean”, with a few wrinkles. It is the default approach used by the emmeans package for R.

Roughly speaking, the procedure is as follows:

  1. Create a prediction grid with one cell for each combination of categorical predictors in the model, and all numeric variables held at their means.
  2. Make adjusted predictions in each cell of the prediction grid.
  3. Take the average of those predictions (marginal means) for each combination of btype (focal variable) and resp (group by variable).
  4. Compute pairwise differences (contrasts) in marginal means across different levels of the focal variable btype.

The contrast obtained through this approach has two critical characteristics:

  1. It is the contrast for a synthetic individual with perfectly average qualities on every (numeric) predictor.
  2. It is a weighted average of unit-level contrasts, where weights assume a perfectly balanced dataset across every categorical predictor.

With respect to (a), the analyst should ask themselves: Is my quantity of interest the contrast for a perfectly average hypothetical individual? With respect to (b), the analyst should ask themselves: Is my quantity of interest the contrast in a model estimated using (potentially) unbalanced data, but interpreted as if the data were perfectly balanced?

For example, imagine that one of the control variables in your model is a variable measuring educational attainment in 4 categories: No high school, High school, Some college, Completed college. The contrast between marginal is a weighted average of contrasts estimated in the 4 cells, and each of those contrasts will be weighted equally in the overall estimate. If the population of interest is highly unbalanced in the educational categories, then the estimate computed in this way will not be most useful.

If the contrasts between marginal means is really the quantity of interest, it is easy to use the comparisons() to estimate contrasts between marginal means. The newdata determines the values of the predictors at which we want to compute contrasts. We can set newdata="marginalmeans" to emulate the emmeans behavior. For example, here we compute contrasts in a model with an interaction:

Which is equivalent to this in emmeans:

The emmeans section of the Alternative Software vignette shows further examples.

The excellent vignette of the emmeans package discuss the same issues in a slightly different (and more positive) way:

The point is that the marginal means of cell.means give equal weight to each cell. In many situations (especially with experimental data), that is a much fairer way to compute marginal means, in that they are not biased by imbalances in the data. We are, in a sense, estimating what the marginal means would be, had the experiment been balanced. Estimated marginal means (EMMs) serve that need.

All this said, there are certainly situations where equal weighting is not appropriate. Suppose, for example, we have data on sales of a product given different packaging and features. The data could be unbalanced because customers are more attracted to some combinations than others. If our goal is to understand scientifically what packaging and features are inherently more profitable, then equally weighted EMMs may be appropriate; but if our goal is to predict or maximize profit, the ordinary marginal means provide better estimates of what we can expect in the marketplace.

Adjusted Risk ratios

The transform_pre and transform_post arguments of the comparisons() function can be used to compute custom and transformed contrasts, such as Adjusted Risk Ratios. See the transformations vignette for details.