In a previous vignette, we introduced the “marginal effect” as a partial derivative. Since derivatives are only properly defined for continuous variables, we cannot use them to interpret the effects of changes in categorical variables. For this, we turn to contrasts between Adjusted predictions. In the context of this package, a “Contrast” is defined as:
The difference between two adjusted predictions, calculated for meaningfully different regressor values (e.g., College graduates vs. Others).
The marginaleffects()
function automatically calculates contrasts instead of derivatives for factor, logical, or character variables.
The comparisons()
function gives users more powerful features to compute different contrasts. For example, it allows users to create custom contrasts like ratios instead of differences, and with different values of the predictors.
Consider a simple model with a logical and a factor variable:
library(marginaleffects)
library(magrittr)
tmp <- mtcars
tmp$am <- as.logical(tmp$am)
mod <- lm(mpg ~ am + factor(cyl), tmp)
The marginaleffects
function automatically computes contrasts for each level of the categorical variables, relative to the baseline category (FALSE
for logicals, and the reference level for factors), while holding all other values at their mode or mean:
mfx <- marginaleffects(mod)
summary(mfx)
#> Average marginal effects
#> Term Contrast Effect Std. Error z value Pr(>|z|) 2.5 % 97.5 %
#> 1 am TRUE - FALSE 2.560 1.298 1.973 0.04851 0.01675 5.103
#> 2 cyl 6 - 4 -6.156 1.536 -4.009 6.1077e-05 -9.16608 -3.146
#> 3 cyl 8 - 4 -10.068 1.452 -6.933 4.1147e-12 -12.91359 -7.222
#>
#> Model type: lm
#> Prediction type: response
The summary printed above says that moving from the reference category 4
to the level 6
on the cyl
factor variable is associated with a change of -6.156 in the adjusted prediction. Similarly, the contrast from FALSE
to TRUE
on the am
variable is equal to 2.560.
We can obtain different contrasts by using the comparisons()
function. For example:
comparisons(mod, contrast_factor = "sequential") %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response am TRUE - FALSE 2.559954 1.297579 1.972869 4.851044e-02
#> 2 response cyl 6 - 4 -6.156118 1.535723 -4.008612 6.107658e-05
#> 3 response cyl 8 - 6 -3.911442 1.470254 -2.660385 7.805144e-03
#> conf.low conf.high
#> 1 0.01674586 5.103162
#> 2 -9.16607927 -3.146156
#> 3 -6.79308707 -1.029797
comparisons(mod, contrast_factor = "pairwise") %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response am TRUE - FALSE 2.559954 1.297579 1.972869 4.851044e-02
#> 2 response cyl 6 - 4 -6.156118 1.535723 -4.008612 6.107658e-05
#> 3 response cyl 8 - 4 -10.067560 1.452082 -6.933187 4.114709e-12
#> 4 response cyl 8 - 6 -3.911442 1.470254 -2.660385 7.805144e-03
#> conf.low conf.high
#> 1 0.01674586 5.103162
#> 2 -9.16607927 -3.146156
#> 3 -12.91358877 -7.221530
#> 4 -6.79308707 -1.029797
comparisons(mod, contrast_factor = "reference") %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response am TRUE - FALSE 2.559954 1.297579 1.972869 4.851044e-02
#> 2 response cyl 6 - 4 -6.156118 1.535723 -4.008612 6.107658e-05
#> 3 response cyl 8 - 4 -10.067560 1.452082 -6.933187 4.114709e-12
#> conf.low conf.high
#> 1 0.01674586 5.103162
#> 2 -9.16607927 -3.146156
#> 3 -12.91358877 -7.221530
For comparison, this code produces the same results using the emmeans
package:
library(emmeans)
emm <- emmeans(mod, specs = "cyl")
contrast(emm, method = "revpairwise")
#> contrast estimate SE df t.ratio p.value
#> cyl6 - cyl4 -6.16 1.54 28 -4.009 0.0012
#> cyl8 - cyl4 -10.07 1.45 28 -6.933 <.0001
#> cyl8 - cyl6 -3.91 1.47 28 -2.660 0.0331
#>
#> Results are averaged over the levels of: am
#> P value adjustment: tukey method for comparing a family of 3 estimates
emm <- emmeans(mod, specs = "am")
contrast(emm, method = "revpairwise")
#> contrast estimate SE df t.ratio p.value
#> TRUE - FALSE 2.56 1.3 28 1.973 0.0585
#>
#> Results are averaged over the levels of: cyl
Note that these commands also work on for other types of models, such as GLMs, on different scales:
mod_logit <- glm(am ~ factor(gear), data = mtcars, family = binomial)
comparisons(mod_logit) %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response gear 4 - 3 0.6666667 1.360805e-01 4.899061 9.629569e-07
#> 2 response gear 5 - 3 1.0000000 1.071403e-05 93335.529606 0.000000e+00
#> conf.low conf.high
#> 1 0.3999538 0.9333795
#> 2 0.9999790 1.0000210
comparisons(mod_logit, type = "link") %>% tidy()
#> type term contrast estimate std.error statistic p.value conf.low
#> 1 link gear 4 - 3 21.25922 4577.962 0.004643817 0.9962948 -8951.381
#> 2 link gear 5 - 3 41.13214 9155.924 0.004492407 0.9964156 -17904.148
#> conf.high
#> 1 8993.90
#> 2 17986.41
We can also compute contrasts for differences in numeric variables. For example, we can see what happens to the adjusted predictions when we increment the hp
variable by 1 unit (default) or by 5 units:
mod <- lm(mpg ~ hp, data = mtcars)
comparisons(mod) %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response hp (x + 1) - x -0.06822828 0.0101193 -6.742389 1.558043e-11
#> conf.low conf.high
#> 1 -0.08806175 -0.04839481
comparisons(mod, contrast_numeric = 5) %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response hp (x + 5) - x -0.3411414 0.05059652 -6.742389 1.558043e-11
#> conf.low conf.high
#> 1 -0.4403087 -0.241974
Compare adjusted predictions for a change in the regressor between two arbitrary values:
comparisons(mod, contrast_numeric = c(90, 110)) %>% tidy()
#> type term contrast estimate std.error statistic p.value conf.low
#> 1 response hp 110 - 90 -1.364566 0.2023861 -6.742389 1.558043e-11 -1.761235
#> conf.high
#> 1 -0.9678961
Compare adjusted predictions when the regressor changes across the interquartile range, across one or two standard deviations about its mean, or from across its full range:
comparisons(mod, contrast_numeric = "iqr") %>% tidy()
#> type term contrast estimate std.error statistic p.value conf.low
#> 1 response hp Q3 - Q1 -5.697061 0.8449619 -6.742389 1.558043e-11 -7.353156
#> conf.high
#> 1 -4.040966
comparisons(mod, contrast_numeric = "sd") %>% tidy()
#> type term contrast estimate std.error statistic
#> 1 response hp (x + sd/2) - (x - sd/2) -4.677926 0.6938085 -6.742389
#> p.value conf.low conf.high
#> 1 1.558043e-11 -6.037766 -3.318087
comparisons(mod, contrast_numeric = "2sd") %>% tidy()
#> type term contrast estimate std.error statistic p.value
#> 1 response hp (x + sd) - (x - sd) -9.355853 1.387617 -6.742389 1.558043e-11
#> conf.low conf.high
#> 1 -12.07553 -6.636174
comparisons(mod, contrast_numeric = "minmax") %>% tidy()
#> type term contrast estimate std.error statistic p.value conf.low
#> 1 response hp Max - Min -19.3086 2.863763 -6.742389 1.558043e-11 -24.92147
#> conf.high
#> 1 -13.69573
In some contexts we would like to know what happens when two (or more) predictors change at the same time. In the marginaleffects
package terminology, this is an “interaction between contrasts.”
For example, consider a model with two factor variables:
What happens if am
increases by 1 unit and cyl
changes from a baseline reference to another level?
cmp <- comparisons(mod, variables = c("cyl", "am"))
summary(cmp)
#> Average contrasts
#> cyl am Effect Std. Error z value Pr(>|z|) 2.5 % 97.5 %
#> 1 4 - 4 (x + 1) - x 5.175 2.053 2.5209 0.0117059 1.151 9.199
#> 2 6 - 4 (x + 1) - x -1.983 2.493 -0.7956 0.4262446 -6.869 2.902
#> 3 8 - 4 (x + 1) - x -7.048 2.731 -2.5805 0.0098656 -12.401 -1.695
#>
#> Model type: lm
#> Prediction type: response
When the variables
argument is used and the model formula includes interactions, the “cross-contrasts” contrasts will automatically be displayed. You can also force comparisons()
to do it by setting interactions=TRUE
and using the variables
argument to specify which variables should be manipulated simultaneously.
This section compares 4 quantities:
The ideas discussed in this section focus on contrasts, but they carry over directly to analogous types of marginal effects.
In models with interactions or non-linear components (e.g., link function), the value of a contrast or marginal effect can depend on the value of all the predictors in the model. As a result, contrasts and marginal effects are fundamentally unit-level quantities. The effect of a 1 unit increase in \(X\) can be different for Mary or John. Every row of a dataset has a different contrast and marginal effect.
The mtcars
dataset has 32 rows, so the comparisons()
function produces 32 contrast estimates:
By default, the marginaleffects()
and comparisons()
functions compute marginal effects and contrasts for every row of the original dataset. These unit-level estimates can be unwieldy and hard to interpret. To help interpretation, the summary()
function computes the “Average Marginal Effect” or “Average Contrast,” by taking the mean of all the unit-level estimates.
summary(cmp)
#> Average contrasts
#> Term Contrast Effect Std. Error z value Pr(>|z|) 2.5 % 97.5 %
#> 1 mpg (x + 1) - x 0.06081 0.01284 4.737 2.1714e-06 0.03565 0.08597
#>
#> Model type: glm
#> Prediction type: response
which is equivalent to:
We could also show the full distribution of contrasts across our dataset with a histogram:
library(ggplot2)
cmp <- comparisons(mod, variables = "gear")
ggplot(cmp, aes(comparison)) +
geom_histogram(bins = 30) +
facet_wrap(~contrast, scale = "free_x") +
labs(x = "Distribution of unit-level contrasts")
This graph display the effect of a change of 1 unit in the mpg
variable, for each individual in the observed data.
An alternative which used to be very common but has now fallen into a bit of disfavor is to compute “Contrasts at the mean.” The idea is to create a “synthetic” or “hypothetical” individual (row of the dataset) whose characteristics are completely average. Then, we compute and report the contrast for this specific hypothetical individual.
This can be achieved by setting newdata="mean"
or to newdata=datagrid()
, both of which fix variables to their means or modes:
comparisons(mod, variables = "mpg", newdata = "mean")
#> rowid type term contrast comparison std.error statistic p.value
#> 1 1 response mpg (x + 1) - x 0.1664787 0.06245542 2.66556 0.007686022
#> conf.low conf.high gear mpg
#> 1 0.04406829 0.288889 3 20.09062
Contrasts at the mean can differ substantially from average contrasts.
The advantage of this approach is that it is very cheap and fast computationally. The disadvantage is that the interpretation is somewhat ambiguous. Often times, there simply does not exist an individual who is perfectly average across all dimensions of the dataset. It is also not clear why the analyst should be particularly interested in the contrast for this one, synthetic, perfectly average individual.
Yet another type of contrast is the “Contrast between marginal means.” This type of contrast is closely related to the “Contrast at the mean”, with a few wrinkles. It is the default approach used by the emmeans
package for R
.
Roughly speaking, the procedure is as follows:
btype
(focal variable) and resp
(group by
variable).btype
.The contrast obtained through this approach has two critical characteristics:
With respect to (a), the analyst should ask themselves: Is my quantity of interest the contrast for a perfectly average hypothetical individual? With respect to (b), the analyst should ask themselves: Is my quantity of interest the contrast in a model estimated using (potentially) unbalanced data, but interpreted as if the data were perfectly balanced?
For example, imagine that one of the control variables in your model is a variable measuring educational attainment in 4 categories: No high school, High school, Some college, Completed college. The contrast between marginal is a weighted average of contrasts estimated in the 4 cells, and each of those contrasts will be weighted equally in the overall estimate. If the population of interest is highly unbalanced in the educational categories, then the estimate computed in this way will not be most useful.
If the contrasts between marginal means is really the quantity of interest, it is easy to use the comparisons()
to estimate contrasts between marginal means. The newdata
determines the values of the predictors at which we want to compute contrasts. We can set newdata="marginalmeans"
to emulate the emmeans
behavior. For example, here we compute contrasts in a model with an interaction:
dat <- read.csv("https://vincentarelbundock.github.io/Rdatasets/csv/palmerpenguins/penguins.csv")
mod <- lm(bill_length_mm ~ species * sex + island + body_mass_g, data = dat)
cmp <- comparisons(
mod,
newdata = "marginalmeans",
variables = c("species", "island"))
summary(cmp)
#> Average contrasts
#> species island Effect Std. Error z value Pr(>|z|)
#> 1 Adelie - Adelie Dream - Biscoe -0.45571 0.4533 -1.005 0.31472
#> 2 Adelie - Adelie Torgersen - Biscoe 0.08507 0.4701 0.181 0.85639
#> 3 Chinstrap - Adelie Biscoe - Biscoe 10.26934 0.4067 25.252 < 2.22e-16
#> 4 Chinstrap - Adelie Dream - Biscoe 9.81362 0.4336 22.630 < 2.22e-16
#> 5 Chinstrap - Adelie Torgersen - Biscoe 10.35441 0.6217 16.656 < 2.22e-16
#> 6 Gentoo - Adelie Biscoe - Biscoe 5.89568 0.6773 8.705 < 2.22e-16
#> 7 Gentoo - Adelie Dream - Biscoe 5.43996 0.9413 5.779 7.504e-09
#> 8 Gentoo - Adelie Torgersen - Biscoe 5.98075 0.9542 6.268 3.667e-10
#> 2.5 % 97.5 %
#> 1 -1.3441 0.4327
#> 2 -0.8363 1.0064
#> 3 9.4723 11.0664
#> 4 8.9637 10.6636
#> 5 9.1360 11.5728
#> 6 4.5683 7.2231
#> 7 3.5951 7.2849
#> 8 4.1105 7.8510
#>
#> Model type: lm
#> Prediction type: response
Which is equivalent to this in emmeans
:
emm <- emmeans(
mod,
specs = c("species", "island"))
contrast(emm, method = "trt.vs.ctrl1")
#> contrast estimate SE df t.ratio p.value
#> Chinstrap Biscoe - Adelie Biscoe 10.2693 0.407 324 25.252 <.0001
#> Gentoo Biscoe - Adelie Biscoe 5.8957 0.677 324 8.705 <.0001
#> Adelie Dream - Adelie Biscoe -0.4557 0.453 324 -1.005 0.8274
#> Chinstrap Dream - Adelie Biscoe 9.8136 0.434 324 22.630 <.0001
#> Gentoo Dream - Adelie Biscoe 5.4400 0.941 324 5.779 <.0001
#> Adelie Torgersen - Adelie Biscoe 0.0851 0.470 324 0.181 0.9994
#> Chinstrap Torgersen - Adelie Biscoe 10.3544 0.622 324 16.656 <.0001
#> Gentoo Torgersen - Adelie Biscoe 5.9808 0.954 324 6.268 <.0001
#>
#> Results are averaged over the levels of: sex
#> P value adjustment: dunnettx method for 8 tests
The emmeans
section of the Alternative Software vignette shows further examples.
The excellent vignette of the emmeans
package discuss the same issues in a slightly different (and more positive) way:
The point is that the marginal means of cell.means give equal weight to each cell. In many situations (especially with experimental data), that is a much fairer way to compute marginal means, in that they are not biased by imbalances in the data. We are, in a sense, estimating what the marginal means would be, had the experiment been balanced. Estimated marginal means (EMMs) serve that need.
All this said, there are certainly situations where equal weighting is not appropriate. Suppose, for example, we have data on sales of a product given different packaging and features. The data could be unbalanced because customers are more attracted to some combinations than others. If our goal is to understand scientifically what packaging and features are inherently more profitable, then equally weighted EMMs may be appropriate; but if our goal is to predict or maximize profit, the ordinary marginal means provide better estimates of what we can expect in the marketplace.
The transform_pre
and transform_post
arguments of the comparisons()
function can be used to compute custom and transformed contrasts, such as Adjusted Risk Ratios. See the transformations vignette for details.