Technical notes

Standard Error of the Average Marginal Effect

The summary() and tidy() functions compute the average marginal effect (or contrast) when they are applied to an object produced by marginaleffects() (or comparisons()). This is done in 3 steps:

  1. Extract the Jacobian used to compute unit-level standard errors.
  2. Take the average of that Jacobian.
  3. Estimate the standard error of the average marginal effect as the square root of the diagonal of J’VJ, where V is the variance-covariance matrix.

As explained succinctly on Stack Exchange:

we want the variance of the Average Marginal Effect (AME) and hence our transformed function is: \(AME = \frac{1}{N} \sum_{i=1}^N g_i(x_i,\hat{\beta})\) Then using the delta method we have \(Var \left( g(\hat{\beta}) \right) = J_g' \Omega_{\hat{\beta}} J_g\) where \(\Omega_{\hat{\beta}} = Var(\hat{\beta})\) and \(J_g' = \frac{\partial\left[\frac{1}{N}\sum_{i=1}^N g (x_i,\hat{\beta})\right]}{\partial \hat\beta} = \frac{1}{N}{\left[\sum_{i=1}^N \frac{\partial \left (g (x_i,\hat{\beta})\right)}{\partial \hat\beta}\right]}\) Which justifies using the “average Jacobian” in the delta method to calculate variance of the AME.

References:

Contrasts and Marginal Effects: “Unit-Level” vs. “Average” vs. “At Mean” vs. “Marginal Means”

This section compares 4 quantities:

  1. Unit-Level Contrasts
  2. Average Contrast
  3. Contrast at the Mean
  4. Contrast Between Marginal Means

The ideas discussed in this section focus on contrasts, but they carry over directly to analogous types of marginal effects.

Unit-level contrasts

In models with interactions or non-linear components (e.g., link function), the value of a contrast or marginal effect can depend on the value of all the predictors in the model. As a result, contrasts and marginal effects are fundamentally unit-level quantities. The effect of a 1 unit increase in \(X\) can be different for Mary or John. Every row of a dataset has a different contrast and marginal effect.

The mtcars dataset has 32 rows, so the comparisons() function produces 32 contrast estimates:

Average contrasts

By default, the marginaleffects() and comparisons() functions compute marginal effects and contrasts for every row of the original dataset. These unit-level estimates can be unwieldy and hard to interpret. To help interpretation, the summary() function computes the “Average Marginal Effect” or “Average Contrast,” by taking the mean of all the unit-level estimates.

which is equivalent to:

We could also show the full distribution of contrasts across our dataset with a histogram:

This graph display the effect of a change of 1 unit in the mpg variable, for each individual in the observed data.

Contrasts at the mean

An alternative which used to be very common but has now fallen into a bit of disfavor is to compute “Contrasts at the mean.” The idea is to create a “synthetic” or “hypothetical” individual (row of the dataset) whose characteristics are completely average. Then, we compute and report the contrast for this specific hypothetical individual.

This can be achieved by calling using the newdata argument and the datagrid() function, which fixes variables to their mean by default:

Contrasts at the mean can differ substantially from average contrasts.

The advantage of this approach is that it is very cheap and fast computationally. The disadvantage is that the interpretation is somewhat ambiguous. Often times, there simply does not exist an individual who is perfectly average across all dimensions of the dataset. It is also not clear why the analyst should be particularly interested in the contrast for this one, synthetic, perfectly average individual.

Contrasts between marginal means

Yet another type of contrast is the “Contrast between marginal means.” This type of contrast is closely related to the “Contrast at the mean”, with a few wrinkles. It is the default approach used by the emmeans package for R.

Roughly speaking, the procedure is as follows:

  1. Create a prediction grid with one cell for each combination of categorical predictors in the model, and all numeric variables held at their means.
  2. Make adjusted predictions in each cell of the prediction grid.
  3. Take the average of those predictions (marginal means) for each combination of btype (focal variable) and resp (group by variable).
  4. Compute pairwise differences (contrasts) in marginal means across different levels of the focal variable btype.

The contrast obtained through this approach has two critical characteristics:

  1. It is the contrast for a synthetic individual with perfectly average qualities on every (numeric) predictor.
  2. It is a weighted average of unit-level contrasts that assumes a perfectly balanced experiment.

With respect to (a), the analyst should ask themselves: Is my quantity of interest the contrast for a perfectly average hypothetical individual? With respect to (b), the analyst should ask themselves: Is my quantity of interest the contrast in a model estimated using unbalanced data, but interpreted as if the data were perfectly balanced?

If the answer to both questions is “yes”, then it is easy to use the comparisons() to estimate contrasts between marginal means. The emmeans section of the Alternative Software vignette shows several examples with parallel code examples to highlight parallels between marginaleffects and emmeans.

The excellent vignette of the emmeans package discuss the same issue in a slightly different way:

The point is that the marginal means of cell.means give equal weight to each cell. In many situations (especially with experimental data), that is a much fairer way to compute marginal means, in that they are not biased by imbalances in the data. We are, in a sense, estimating what the marginal means would be, had the experiment been balanced. Estimated marginal means (EMMs) serve that need.

All this said, there are certainly situations where equal weighting is not appropriate. Suppose, for example, we have data on sales of a product given different packaging and features. The data could be unbalanced because customers are more attracted to some combinations than others. If our goal is to understand scientifically what packaging and features are inherently more profitable, then equally weighted EMMs may be appropriate; but if our goal is to predict or maximize profit, the ordinary marginal means provide better estimates of what we can expect in the marketplace.