Diagnostics
Diagnostics are also an important part of deciding whether the weights perform well. There are several areas that we will explore:
- Effective sample size
- Mean balance
- Distributional balance
1. Effective sample size
Typically, estimated samples sizes with weights are calculated as
\(\sum_i 1/w_i^2\) and gives us a
measure of how much information is in the sample. The lower the
effective sample size (ESS), the higher the variance, and the lower the
sample size, the more weight a few individuals have. Of course, we can
calculate this in causalOT!
Of course, this measure has problems because it can fail to diagnose
problems with variable weights. In response, Vehtari et al. use Pareto
smoothed importance sampling. We offer some shell code to adapt the
class causalWeights to the loo package:
This will also return the Pareto smoothed weights and log weights.
If we want to easily examine the PSIS diagnostics, we can pull those out too
PSIS_diag(raw_psis)
#> $w0
#> $w0$pareto_k
#> [1] -0.2320877
#>
#> $w0$n_eff
#> [1] 115.7568
#>
#>
#> $w1
#> $w1$pareto_k
#> [1] 0.2478191
#>
#> $w1$n_eff
#> [1] 82.99825We can see that all of the \(k\)
values are below the recommended 0.5, indicating finite variance and
that the central limit theorem holds. Note the estimated sample sizes
are a bit lower than the ESS method above.
2. Mean balance
Many authors consider the standardized absolute mean balance as a
marker for important balance: see Stuart
(2010). That is \[ \frac{|\overline{X}_c
- \overline{X}_t| }{\sigma_{\text{pool}}},\] where \(\overline{X}_c\) is the mean in the
controls, \(\overline{X}_t\) is the
mean in the treated, and \(\sigma_{\text{pool}}\) is the pooled
standard deviation. We offer such checks in causalOT as
well.
First, we consider pre-weighting mean balance between treatment groups
mean_balance(x = hainmueller)
#> X1 X2 X3 X4 X5 X6
#> 1.0889178 0.9320099 0.9322327 0.3741322 0.2798986 0.2644929and after weighting mean balance between treatment groups
mean_balance(x = hainmueller, weights = weights)
#> X1 X2 X3 X4 X5 X6
#> 0.036900048 0.006577199 0.003137536 0.003228830 0.002445118 0.060551316Pretty good! However, mean balance doesn’t ensure distributional balance.
3. Distributional balance
Ultimately, distributional balance is what we care about in causal inference. Fortunately, we can also measure that too. We consider the 2-Sinkhorn divergence of Genevay et al. since it metrizes the convergence in distribution.
Before weighting, distributional balance looks poor:
# controls
ot_distance(x1 = hainmueller$get_x0(), x2 = hainmueller$get_x(),
a = NULL, b = rep(1/512,512),
p = 2, penalty = 1e3, debias = TRUE)
#> [1] 0.5311378
#treated
ot_distance(x1 = hainmueller$get_x1(), x2 = hainmueller$get_x(),
a = NULL, b = rep(1/512,512),
p = 2, penalty = 1e3, debias = TRUE)
#> [1] 0.4612781But after weighting, it looks much better!
# controls
ot_distance(x1 = hainmueller$get_x0(), x2 = hainmueller$get_x(),
a = weights@w0, b = rep(1/512,512),
p = 2, penalty = 1e3, debias = TRUE)
#> [1] 0.003670398
# treated
ot_distance(x1 = hainmueller$get_x1(), x2 = hainmueller$get_x(),
a = weights@w1, b = rep(1/512,512),
p = 2, penalty = 1e3, debias = TRUE)
#> [1] 0.002123788After Causal Optimal Transport, the distributions are much
more similar. We can also simply feed the output of calc_weight directly
into the ot_distance function:
ot_distance(x1 = weights, p = 2, penalty = 1e3, debias = TRUE)
#> $pre
#> control treated
#> 0.5311378 0.4612781
#>
#> $post
#> control treated
#> 0.003670398 0.002123788and the S4 deployment takes care of the rest.
Finally, we can construct a summary of the optimal transport distances, Pareto k statistics, effective sample size, and mean balance using the summary method:
We can then print the object to the screen:
summarized_cw
#> Diagnostics for causalWeights for estimand ATE
#> Control group
#> pre post
#> OT distance 0.5311378 0.003670398
#> Pareto k NA -0.232087710
#> N eff 247.0000000 115.756836449
#> Avg. std. mean balance 0.3067516 0.017646355
#>
#> Treated group
#> pre post
#> OT distance 0.4612781 2.123788e-03
#> Pareto k NA 2.478191e-01
#> N eff 265.0000000 8.299825e+01
#> Avg. std. mean balance 0.2859156 9.871006e-04or we can make some diagnostic plots too!
