Comparisons to alternative software

If you do not like marginaleffects, you may want to consider one of the alternatives described below:

emmeans

The emmeans package is developed by Russell V. Lenth and colleagues. It is an extremely powerful package which focuses on the computation of marginal means, but which can also compute slopes.

In my (Vincent’s) biased opinion, the main benefits of marginaleffects over emmeans:

Many of marginaleffects advantages come down to subjective preferences over user interface. Readers are thus encouraged to try both packages to see which interface they prefer. Please keep in mind that emmeans offers many features which are not yet available in marginaleffects, including:

The Marginal Means Vignette includes side-by-side comparisons of emmeans and marginaleffects to compute marginal means. The rest of this section compares the syntax for contrasts and marginaleffects.

Contrasts

AFAICT, emmeans does not provide an easy way to compute unit-level contrasts for every row of the dataset used to fit our model. Therefore, the side-by-side syntax shown below will always include newdata=datagrid() to specify that we want to compute only one contrast: at the mean values of the regressors. In day-to-day practice with marginaleffects(), however, this extra argument would not be necessary.

Fit a model:

Response scale, reference groups:

Link scale, pairwise contrasts:

Contrasts by group

Here is a slightly more complicated example with contrasts estimated by subgroup in a glmmTMB mixed effects model. First we estimate a model and compute pairwise contrasts by subgroup using emmeans:

What did emmeans do to obtain these results? Roughly speaking:

  1. Create a prediction grid with one cell for each combination of categorical predictor in the model.
  2. Make adjusted predictions in each cell of the prediction grid.
  3. Take the average of those predictions (marginal means) for each combination of btype and resp, the focal variable, and the group variable specified in the by argument.
  4. Compute pairwise differences (contrasts) in marginal means across different levels of the focal variable btype.

In short, emmeans computes pairwise contrasts between marginal means. This is different from the default types of contrasts produced by comparisons(), which does not average across a pre-specified grid of predictors. What does comparisons() do instead?

Let newdata be a data frame supplied by the user (or the original data frame used to fit the model), then:

  1. Create a new data frame called newdata2, which is identical to newdata except that the focal variable is incremented by one level.
  2. Compute contrasts as the difference between adjusted predictions made on the two datasets:
    • predict(model, newdata = newdata2) - predict(model, newdata = newdata)

Although it is not idiomatic, we can use still use comparisons() to emulate the emmeans results. First, we create a prediction grid with one cell for each combination of categorical predictor in the model:

This grid has 18 rows, one for each combination of levels for the resp (3), situ (2), and btype (3) variables (3 * 2 * 3 = 18).

Then we compute pairwise contrasts over this grid:

There are 3 pairwise contrasts, corresponding to the 3 pairwise comparisons possible between the 3 levels of the focal variable btype: scold-curse, shout-scold, shout-curse. The comparisons() estimates contrasts for each row of newdata, so we get \(18 \times 3 = 54\) rows.

Finally, we wanted contrasts for each subgroup of the resp variable, so we average out the contrasts using the summary() and the by argument:

These results are identical to those produced by emmeans (except for \(t\) vs. \(z\)).

Marginal Effects

AFAICT, emmeans::emtrends makes it easier to compute marginal effects for a few user-specified values than for large grids or for the full original dataset.

Response scale, user-specified values:

Link scale, user-specified values:

margins and prediction

The margins and prediction packages for R were designed by Thomas Leeper to emulate the behavior of the margins command from Stata. These packages are trailblazers and strongly influenced the development of marginaleffects. The main benefits of marginaleffects over these packages are:

The syntax of the two packages is very similar.

Average Marginal Effects

Individual-Level Marginal Effects

Marginal effects in a user-specified data frame:

head(data.frame(mar))
#>    mpg cyl disp  hp drat    wt  qsec vs am gear carb   fitted se.fitted   dydx_cyl    dydx_hp
#> 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4 22.82043 0.6876212 -0.9416168 -0.0180381
#> 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4 22.01285 0.6056817 -0.9416168 -0.0180381
#> 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1 25.96040 0.7349593 -0.9416168 -0.0180381
#> 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1 20.93608 0.5800910 -0.9416168 -0.0180381
#> 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2 17.16780 0.8322986 -0.9416168 -0.0180381
#> 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1 20.25036 0.6638322 -0.9416168 -0.0180381
#>     dydx_wt Var_dydx_cyl  Var_dydx_hp Var_dydx_wt X_weights X_at_number
#> 1 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1
#> 2 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1
#> 3 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1
#> 4 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1
#> 5 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1
#> 6 -3.166973    0.3035093 0.0001410454   0.5484543        NA           1

head(mfx)
#>   rowid     type term       dydx std.error statistic    p.value  conf.low conf.high  mpg cyl  hp
#> 1     1 response  cyl -0.9416168 0.5509164 -1.709183 0.08741712 -2.021393 0.1381595 21.0   6 110
#> 2     2 response  cyl -0.9416168 0.5509164 -1.709183 0.08741712 -2.021393 0.1381595 21.0   6 110
#> 3     3 response  cyl -0.9416168 0.5509160 -1.709184 0.08741689 -2.021392 0.1381588 22.8   4  93
#> 4     4 response  cyl -0.9416168 0.5509164 -1.709183 0.08741712 -2.021393 0.1381595 21.4   6 110
#> 5     5 response  cyl -0.9416168 0.5509167 -1.709182 0.08741729 -2.021394 0.1381601 18.7   8 175
#> 6     6 response  cyl -0.9416168 0.5509164 -1.709183 0.08741712 -2.021393 0.1381595 18.1   6 105
#>      wt
#> 1 2.620
#> 2 2.875
#> 3 2.320
#> 4 3.215
#> 5 3.440
#> 6 3.460
nd <- data.frame(cyl = 4, hp = 110, wt = 3)

Marginal Effects at the Mean

Counterfactual Average Marginal Effects

The at argument of the margins package emulates Stata by fixing the values of some variables at user-specified values, and by replicating the full dataset several times for each combination of the supplied values (see the Stata section below). For example, if the dataset includes 32 rows and the user calls at=list(cyl=c(4, 6)), margins will compute 64 unit-level marginal effects estimates:

dat <- mtcars
dat$cyl <- factor(dat$cyl)
mod <- lm(mpg ~ cyl * hp + wt, data = mtcars)

mar <- margins(mod, at = list(cyl = c(4, 6, 8)))
summary(mar)
#>  factor    cyl     AME     SE       z      p   lower   upper
#>     cyl 4.0000  0.0381 0.5999  0.0636 0.9493 -1.1376  1.2139
#>     cyl 6.0000  0.0381 0.5999  0.0636 0.9493 -1.1376  1.2138
#>     cyl 8.0000  0.0381 0.5999  0.0636 0.9493 -1.1376  1.2139
#>      hp 4.0000 -0.0878 0.0267 -3.2937 0.0010 -0.1400 -0.0355
#>      hp 6.0000 -0.0499 0.0154 -3.2397 0.0012 -0.0800 -0.0197
#>      hp 8.0000 -0.0120 0.0108 -1.1065 0.2685 -0.0332  0.0092
#>      wt 4.0000 -3.1198 0.6613 -4.7175 0.0000 -4.4160 -1.8236
#>      wt 6.0000 -3.1198 0.6613 -4.7175 0.0000 -4.4160 -1.8236
#>      wt 8.0000 -3.1198 0.6613 -4.7176 0.0000 -4.4160 -1.8237

mfx <- marginaleffects(mod, newdata = datagrid(cyl = c(4, 6, 8), grid.type = "counterfactual"))
summary(mfx, by = "cyl")
#> Average marginal effects 
#>   Term cyl   Effect Std. Error  z value   Pr(>|z|)    2.5 %    97.5 %
#> 1  cyl   4  0.03814    0.59989  0.06357 0.94930949 -1.13762  1.213899
#> 2  cyl   6  0.03814    0.59989  0.06357 0.94930949 -1.13762  1.213899
#> 3  cyl   8  0.03814    0.59989  0.06357 0.94930949 -1.13762  1.213899
#> 4   hp   4 -0.08778    0.02665 -3.29366 0.00098891 -0.14001 -0.035545
#> 5   hp   6 -0.04987    0.01539 -3.23974 0.00119638 -0.08004 -0.019701
#> 6   hp   8 -0.01197    0.01081 -1.10649 0.26851651 -0.03316  0.009229
#> 7   wt   4 -3.11981    0.66132 -4.71754 2.3871e-06 -4.41598 -1.823648
#> 8   wt   6 -3.11981    0.66132 -4.71754 2.3871e-06 -4.41598 -1.823648
#> 9   wt   8 -3.11981    0.66132 -4.71754 2.3871e-06 -4.41598 -1.823648
#> 
#> Model type:  lm 
#> Prediction type:  response

Adjusted Presdictions

The syntax to compute adjusted predictions using the predictions package or marginaleffects is very similar:

Stata

Stata is a good but expensive software package for statistical analysis. It is published by StataCorp LLC. This section compares Stata’s margins command to marginaleffects.

The results produced by marginaleffects are extensively tested against Stata. See the test suite for a list of the dozens of models where we compared estimates and standard errors.

Average Marginal Effect (AMEs)

Marginal effects are unit-level quantities. To compute “average marginal effects”, we first calculate marginal effects for each observation in a dataset. Then, we take the mean of those unit-level marginal effects.

Stata

Both Stata’s margins command and the marginaleffects function can calculate average marginal effects (AMEs). Here is an example showing how to estimate AMEs in Stata:

quietly reg mpg cyl hp wt
margins, dydx(*)

Average marginal effects                        Number of obs     =         32
Model VCE    : OLS
 
Expression   : Linear prediction, predict()
dy/dx w.r.t. : cyl hp wt
 
------------------------------------------------------------------------------
    |            Delta-method
    |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------------------------------------------------------------------
cyl |  -.9416168   .5509164    -1.71   0.098    -2.070118    .1868842
 hp |  -.0180381   .0118762    -1.52   0.140    -.0423655    .0062893
 wt |  -3.166973   .7405759    -4.28   0.000    -4.683974   -1.649972
------------------------------------------------------------------------------

marginaleffects

The same results can be obtained with marginaleffects() and summary() like this:

Note that Stata reports t statistics while marginaleffects reports Z. This produces slightly different p-values because this model has low degrees of freedom: mtcars only has 32 rows

Counterfactual Marginal Effects

A “counterfactual marginal effect” is a special quantity obtained by replicating a dataset while fixing some regressor to user-defined values.

Concretely, Stata computes counterfactual marginal effects in 3 steps:

  1. Duplicate the whole dataset 3 times and sets the values of cyl to the three specified values in each of those subsets.
  2. Calculate marginal effects for each observation in that large grid.
  3. Take the average of marginal effects for each value of the variable of interest.

Stata

With the at argument, Stata’s margins command estimates average counterfactual marginal effects. Here is an example:

quietly reg mpg i.cyl##c.hp wt
margins, dydx(hp) at(cyl = (4 6 8))

Average marginal effects                        Number of obs     =         32
Model VCE    : OLS

Expression   : Linear prediction, predict()
dy/dx w.r.t. : hp

1._at        : cyl             =           4

2._at        : cyl             =           6

3._at        : cyl             =           8

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
hp           |
         _at |
          1  |   -.099466   .0348665    -2.85   0.009    -.1712749   -.0276571
          2  |  -.0213768    .038822    -0.55   0.587    -.1013323    .0585787
          3  |   -.013441   .0125138    -1.07   0.293    -.0392137    .0123317
------------------------------------------------------------------------------

Average Counterfactual Adjusted Predictions

Stata

Just like Stata’s margins command computes average counterfactual marginal effects, it can also estimate average counterfactual adjusted predictions.

Here is an example:

quietly reg mpg i.cyl##c.hp wt
margins, at(cyl = (4 6 8))

Predictive margins                              Number of obs     =         32
Model VCE    : OLS

Expression   : Linear prediction, predict()

1._at        : cyl             =           4

2._at        : cyl             =           6

3._at        : cyl             =           8

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         _at |
          1  |   17.44233   2.372914     7.35   0.000     12.55522    22.32944
          2  |    18.9149   1.291483    14.65   0.000     16.25505    21.57476
          3  |   18.33318   1.123874    16.31   0.000     16.01852    20.64785
------------------------------------------------------------------------------

Again, this is what Stata does in the background:

  1. It duplicates the whole dataset 3 times and sets the values of cyl to the three specified values in each of those subsets.
  2. It calculates predictions for that large grid.
  3. It takes the average prediction for each value of cyl.

In other words, average counterfactual adjusted predictions as implemented by Stata are a hybrid between predictions at the observed values (the default in marginaleffects::predictions) and predictions at representative values.

marginaleffects

You can estimate average counterfactual adjusted predictions with predictions() by, first, setting the grid.type argument of data.grid() to "counterfactual" and, second, by averaging the predictions, for example using aggregate() or dplyr::summarise().

Note that standard errors are not yet available for average adjusted predictions.

brmsmargins

The brmsmargins package is developed by Joshua Wiley:

This package has functions to calculate marginal effects from brms models ( http://paul-buerkner.github.io/brms/ ). A central motivator is to calculate average marginal effects (AMEs) for continuous and discrete predictors in fixed effects only and mixed effects regression models including location scale models.

The two main advantages of brmsmargins over marginaleffects are:

  1. Ability to compute “Marginal Coefficients” following the method described in Hedeker et al (2012).
  2. Ability to “integrate out random effects.”

The main advantages of marginaleffects over brmsmargins are:

  1. Support for 60+ model types, rather than just the brms package.
  2. Simpler user interface (subjective).

The rest of this section presents side-by-side replications of some of the analyses from the brmsmargins vignettes in order to show highlight parallels and differences in syntax.

Marginal Effects for Fixed Effects Models

Marginal Effects for Mixed Effects Models

Estimate a mixed effects logistic regression model with brms:

Marginal Effects for Location Scale Models

effects

The effects package was created by John Fox and colleagues.

effects offers several options which are not currently available in marginaleffects, including:

modelbased

The modelbased package is developed by the easystats team.

This section is incomplete; contributions are welcome.

ggeffects

The ggeffects package is developed by Daniel Lüdecke.

This section is incomplete; contributions are welcome.