ALE effect size measures
ALE was developed to graphically display the relationship between predictor variables in a model and the outcome regardless of the nature of the model. Thus, before we proceed to describe our extension of effect size measures based on ALE, let us first briefly examine the ALE plots for each variable.
ALE plots with p-values
We can see that most variables seem to have some sort of mean effect across various values. However, for statistical inference, our focus must be on the bootstrap intervals. Crucial to our interpretation is the middle grey band that indicates the median ± 5% of random values. Below, we will explain what exactly the ALE range (ALER) means, but for now, we can say this:
- The approximate middle of the grey band is the median of the y
outcome variables in the dataset (
math_avg, in our case). The middle tick on the right y axis indicates the exact median. (Themodel_bootstrap()function lets you centre the data on the mean or on zero if you prefer with therelative_yargument.) - We call this grey band the “ALER band”. 95% of random variables had ALE values that fully lay within the ALER band.
- The dashed lines above and below the ALER band expand the boundaries to where 99% of the random variables were constrained. These boundaries could be considered as demarcating an extended or outward ALER band.
The idea is that if the ALE values of any predictor variable falls
fully within the ALER band, then it has no greater effect than 95% of
purely random variables. Moreover, to consider any effect in the ALE
plot to be statistically significant (that is, non-random), there should
be no overlap between the bootstrapped confidence regions of a predictor
variable and the ALER band. (For the threshold p-values, We use the
conventional defaults of 0.05 for 95% confidence and 0.01 for 99%
confidence, but the value can be changed with the p_alpha
argument.)
For categorical variables (public and
high_minority above), the confidence interval bars for all
categories overlap the ALER band. The confidence interval bars indicate
two useful pieces of information to us. When we compare them to the ALER
band, their overlap or lack thereof tells us about the practical
significance of the category. When we compare the confidence bars of one
category with those of others, it allows us to assess if the category
has a statistically significant effect that is different from that of
the other categories; this is equivalent to the regular interpretation
of coefficients for GAM and other GLM models. In both cases, the
confidence interval bars of the TRUE and FALSE categories overlap each
other, indicating that there is no statistically significant difference
between categories. Whereas the coefficient table above based on classic
statistics indicated this conclusion for public, it
indicated that high_minority had a statistically
significant effect; our ALE analysis indicates that
high_minority does not. In addition, each confidence
interval band overlaps the ALER band, indicating that none of the
effects is meaningfully different from random results, either.
For numeric variables, the confidence regions overlap the ALER band
for most of the domains of the predictor variables except for some
regions that we will examine. The extreme points of each variable
(except for discrim and female_ratio) are
usually either slightly below or slightly above the ALER band,
indicating that extreme values have the most extreme effects: math
achievement increases with increasing school size, academic track ratio,
and mean socioeconomic status, whereas it decreases with increasing
minority ratio. The ratio of females and the discrimination climate both
overlap the ALER band for the entirety of their domains, so any apparent
trends are not supported by the data.
Of particular interest is the random variable rand_norm,
whose average ALE appears to show some sort of pattern. However, we note
that the 95% confidence intervals we use mean that if we were to retry
the analysis for twenty different random seeds, we would expect at least
one of the random variables to partially escape the bounds of the ALER
band. We will return below to the implications of random variables in
ALE analysis.
ALE plots without p-values
Before we continue, let us take a brief detour to see what we get if
we run model_bootstrap() without passing it a
p_funs object. This might be because we forget to do so, or
because we want to see quick results without the slow process of first
generating a p_funs object. Let us run
model_bootstrap() again, but this time, without
p-values.
mb_gam_no_p <- model_bootstrap(
math,
gam_math,
# For the GAM model coefficients, show details of all variables, parametric or not
tidy_options = list(parametric = TRUE),
# tidy_options = list(parametric = NULL),
boot_it = 40, # 100 by default but reduced here for a faster demonstration
parallel = 2 # CRAN limit (delete this line on your own computer)
)
mb_gam_no_p$ale$plots |>
patchwork::wrap_plots(ncol = 2)
In the absence of p-values, the {ale} packages uses alternate
visualizations to offer meaningful results, but with somewhat different
interpretations of the middle grey band. Without p-values, we do not
have any point of reference for the ALER statistics, so we use
percentiles around the median as the reference. The middle grey band
here indicates the median ± 2.5%, that is, the middle 5% of all average
mathematics achievement scores (math_avg) values in the
dataset. We call this the “median band”. The idea is that if any
predictor can do no better than influencing math_avg to
fall within this middle median band, then it only has a minimal effect.
For an effect to be considered statistically significant, there should
be no overlap between the confidence regions of a predictor variable and
the median band. (We use 5% around the median by default, but the value
can be changed with the median_band_pct argument.) For
further reference, the outer dashed lines indicate the interquartile
range of the outcome values, that is, the 25th and 75th percentiles.
We can see that in this case, the 5% median band is much narrower than the 5% ALER band when p-values are calculated, though they might be more similar for a different dataset. This should give us pause to skipping the calculation of p-values, since we might be overly lax in interpreting apparent relationships as meaningful whereas the ALER band indicates that they might not be that different from what random variables might produce.
For most of the rest of this article, we will only analyze the results with ALER bands generated from p-values, though we will briefly revisit median bands without p-values.
ALE effect size measures on the scale of the y outcome variable
Although ALE plots allow rapid and intuitive conclusions for
statistical inference, it is often helpful to have summary numbers that
quantify the average strengths of the effects of a variable. Thus, we
have developed a collection of effect size measures based on ALE
tailored for intuitive interpretation. To understand the intuition
underlying the various ALE effect size measures, it is useful to first
examine the ALE effects plot, which graphically
summarizes the effect sizes of all the variables in the ALE analysis.
This is generated when ale is executed and both statistics
and plots are requested (which is the case by default) and is accessible
with the To focus on all the measures for a specific variable, we can
access the ale$stats$effects_plot element:
This plot is unusual, so it requires some explanation:
- The y (vertical) axis displays the x variables, rather than the x axis. This is consistent with most effect size plots because they list the full names of variables. It is more readable to list them as labels on the y axis than the other way around.
- The x (horizontal) axis thus displays the y (outcome) variable. But there are two representations of this same axis, one at the bottom and one at the top.
- On the bottom is a more typical axis of the outcome variable, in our
case,
math_avg. It is scaled as expected. In our case, the axis breaks default to five units each from 5 to 20, evenly spaced. - On the top, the outcome variable is expressed as percentiles ranging from 0 (the minimum outcome value in the dataset) to 100 (the maximum). It is divided into 10 deciles of 10% each. Because percentiles are usually not evenly distributed in a dataset, the decile breaks are not evenly spaced.
- Thus, this plot has two x axes, the lower one in units of the outcome variable and the upper one in percentiles of the outcome variable. To reduce the confusion, the major vertical gridlines that are slightly darker align with the units of the outcome (lower axis) and the minor vertical gridlines that are slightly lighter align with the percentiles (upper axis).
- The vertical grey band in the middle is the NALED band. Its width is the 0.05 p-value of the NALED (explained below). That is, 95% of random variables had a NALED equal or smaller than that width.
- The variables on the horizontal axis are sorted by decreasing ALED and then NALED value (explained below).
Although it is somewhat confusing to have two axes, the percentiles are a direct transformation of the raw outcome values. The first two base ALE effect size measures below are in units of the outcome variable while their normalized versions are in percentiles of the outcome. Thus, the same plot can display the two kinds of measures simultaneously. Referring to this plot can help understand each of the measures, which we proceed to explain in detail.
Before we explain these measures in detail, we must reiterate the timeless reminder that correlation is not causation. So, none of the scores necessarily means that an x variable causes a certain effect on the y outcome; we can only say that the ALE effect size measures indicate associated or related variations between the two variables.
ALE range (ALER)
The easiest ALE statistic to understand is the ALE range (ALER), so
we begin there. It is simply the range from the minimum to the maximum
of any ale_y value for that variable. Mathematically, that
is
\[\mathrm{ALER}(\mathrm{ale\_y}) = \{
\min(\mathrm{ale\_y}), \max(\mathrm{ale\_y}) \}\]
where \(\mathrm{ale\_y}\) is the vector
of ALE y values for a variable.
All the ALE effect size measures are centred on zero so that they are consistent regardless of if the user chooses to centre their plots on zero, the median, or the mean. Specifically,
aler_min: minimum of anyale_yvalue for the variable.aler_max: maximum of anyale_yvalue for the variable.
ALER shows the extreme values of a variable’s effect on the outcome.
In the effects plot above, it is indicated by the extreme ends of the
horizontal bars for each variable. We can access ALE effect size
measures through the ale$stats element of the bootstrap
result object, with multiple views. To focus on all the measures for a
specific variable, we can access the ale$stats$by_term
element.
Let’s focus on public. Here is its ALE plot:
Here are the effect size measures for the categorical
public:
mb_gam_math$ale$stats$by_term$public
#> # A tibble: 6 × 7
#> statistic estimate p.value conf.low median mean conf.high
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 aled 0.378 0 0.129 0.361 0.378 0.771
#> 2 aler_min -0.342 0.352 -0.747 -0.311 -0.342 -0.111
#> 3 aler_max 0.427 0.241 0.137 0.410 0.427 0.803
#> 4 naled 5.48 0.00100 1.12 5.29 5.48 9.89
#> 5 naler_min -4.90 0.563 -9.66 -5 -4.90 0
#> 6 naler_max 6.24 0.405 1.81 6.21 6.24 13.8We see there that public has an ALER of [-0.34, 0.43].
When we consider that the median math score in the dataset is 12.9, this
ALER indicates that the minimum of any ALE y value for
public (when public == TRUE) is -0.34 below
the median. This is shown at the 12.6 mark in the plot above. The
maximum (public == FALSE) is 0.43 above the median, shown
at the 13.3 point above.
The unit for ALER is the same unit as the outcome variable; in our
case, that is math_avg ranging from 2 to 20. No matter what
the average ALE values might be, the ALER quickly shows the minimum and
maximum effects of any value of the x variable on the y variable.
For contrast, let us look at a numeric variable,
academic_ratio:
Here are its ALE effect size measures:
mb_gam_math$ale$stats$by_term$academic_ratio
#> # A tibble: 6 × 7
#> statistic estimate p.value conf.low median mean conf.high
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 aled 0.671 0 0.222 0.650 0.671 1.20
#> 2 aler_min -3.55 0 -7.13 -3.14 -3.55 -0.412
#> 3 aler_max 2.02 0 0.840 2.12 2.02 3.07
#> 4 naled 9.01 0 3.96 8.60 9.01 14.7
#> 5 naler_min -30.1 0 -48.8 -31.6 -30.1 -4.95
#> 6 naler_max 28.0 0 9.23 29.4 28.0 38.8The ALER for academic_ratio is considerably broader with
-3.55 below and 2.02 above the median.
ALE deviation (ALED)
While the ALE range shows the most extreme effects a variable might have on the outcome, the ALE deviation indicates its average effect over its full domain of values. With the zero-centred ALE values, it is conceptually similar to the weighted mean absolute error (MAE) of the ALE y values. Mathematically, it is
\[ \mathrm{ALED}(\mathrm{ale\_y}, \mathrm{ale\_n}) = \frac{\sum_{i=1}^{k} \left| \mathrm{ale\_y}_i \times \mathrm{ale\_n}_i \right|}{\sum_{i=1}^{k} \mathrm{ale\_n}_i} \] where \(i\) is the index of \(k\) ALE x intervals for the variable (for a categorical variable, this is the number of distinct categories), \(\mathrm{ale\_y}_i\) is the ALE y value for the \(i\)th ALE x interval, and \(\mathrm{ale\_n}_i\) is the number of rows of data in the \(i\)th ALE x interval.
Based on its ALED, we can say that the average effect on math scores of whether a school is in the public or Catholic sector is 0.38 (again, out of a range from 2 to 20). In the effects plot above, the ALED is indicated by a white box bounded by parentheses ( and ). As it is centred on the median, we can readily see that the average effect of school sector barely exceeds the limits of the ALER band, indicating that it barely exceeds our threshold of practical relevance. The average effect for ratio of academic track students is slightly higher at 0.67. We can see on the plot that it slightly exceeds the ALER band on both sides, indicating its slightly stronger effect. We will comment on the values of other variables when we discuss the normalized versions of these scores, to which we proceed next.
Normalized ALE effect size measures
Since ALER and ALED scores are scaled on the range of y for a given dataset, these scores cannot be compared across datasets. Thus, we present normalized versions of each with intuitive, comparable values. For intuitive interpretation, we normalize the scores on the minimum, median, and maximum of any dataset. In principle, we divide the zero-centred y values in a dataset into two halves: the lower half from the 0th to the 50th percentile (the median) and the upper half from the 50th to the 100th percentile. (Note that the median is included in both halves). With zero-centred ALE y values, all negative and zero values are converted to their percentile score relative to the lower half of the original y values while all positive ALE y values are converted to their percentile score relative to the upper half. (Technically, this percentile assignment is called the empirical cumulative distribution function (ECDF) of each half.) Each half is then divided by two to scale them from 0 to 50 so that together they can represent 100 percentiles. (Note: when a centred ALE y value of exactly 0 occurs, we choose to include the score of zero ALE y in the lower half because it is analogous to the 50th percentile of all values, which more intuitively belongs in the lower half of 100 percentiles.) The transformed maximum ALE y is then scaled as a percentile from 0 to 100%.
There is a notable complication. This normalization smoothly distributes ALE y values when there are many distinct values, but when there are only a few distinct ALE y values, then even a minimal ALE y deviation can have a relatively large percentile difference. If any ALE y value is less than the difference between the median in the data and the value either immediately below or above the median, we consider that it has virtually no effect. Thus, the normalization sets such minimal ALE y values as zero.
Its formula is:
\[
norm\_ale\_y = 100 \times \begin{cases}
0 & \text{if } \max(centred\_y < 0) \leq ale\_y \leq
\min(centred\_y > 0), \\
\frac{-ECDF_{y_{\leq 0}}(ale\_y)}{2} & \text{if }ale\_y < 0 \\
\frac{ECDF_{y_{\geq 0}}(ale\_y)}{2} & \text{if }ale\_y > 0 \\
\end{cases}
\] where - \(centred\_y\) is the
vector of y values centred on the median (that is, the
median is subtracted from all values). - \(ECDF_{y_{\geq 0}}\) is the ECDF of the
non-negative values in y. - \(-ECDF_{y_{\leq 0}}\) is the ECDF of the
negative values in y after they have been inverted
(multiplied by -1).
Of course, the formula could be simplified by multiplying by 50 instead of by 100 and not dividing the ECDFs by two each. But we prefer the form we have given because it is explicit that each ECDF represents only half the percentile range and that the result is scored to 100 percentiles.
Normalized ALER (NALER)
Based on this normalization, we first have the normalized ALER (NALER), which scales the minimum and maximum ALE y values from -50% to +50%, centred on 0%, which represents the median:
\[ \mathrm{NALER}(\mathrm{y, ale\_y}) = \{\min(\mathrm{norm\_ale\_y}) + 50, \max(\mathrm{norm\_ale\_y}) + 50 \} \]
where \(y\) is the full vector of y values in the original dataset, required to calculate \(\mathrm{norm\_ale\_y}\).
ALER shows the extreme values of a variable’s effect on the outcome.
In the effects plot above, it is indicated by the extreme ends of the
horizontal bars for each variable. We see there that public
has an ALER of -0.34, 0.43. When we consider that the median math score
in the dataset is 12.9, this ALER indicates that the minimum of any ALE
y value for public (when public == TRUE) is
-0.34 below the median. This is shown at the 12.6 mark in the plot
above. The maximum (public == FALSE) is 0.43 above the
median, shown at the 13.3 point above. The ALER for
academic_ratio is considerably broader with -3.55 below and
2.02 above the median.
The result of this transformation is that NALER values can be interpreted as percentile effects of y below or above the median, which is centred at 0%. Their numbers represent the limits of the effect of the x variable with units in percentile scores of y. In the effects plot above, because the percentile scale on the top corresponds exactly to the raw scale below, the NALER limits are represented by exactly the same points as the ALER limits; only the scale changes. The scale for ALER and ALED is the lower scale of the raw outcomes; the scale for NALER and NALED is the upper scale of percentiles.
So, with a NALER of -4.9, 6.24, the minimum of any ALE value for
public (public == TRUE) shifts math scores by
-5 percentile y points whereas the maximum
(public == FALSE) shifts math scores by 6 percentile
points. Academic track ratio has a NALER of -30.08, 27.95, ranging from
-30 to 28 percentile points of math scores.
Normalized ALED (NALED)
The normalization of ALED scores applies the same ALED formula as before but on the normalized ALE values instead of on the original ALE y values:
\[ \mathrm{NALED}(y, \mathrm{ale\_y}, \mathrm{ale\_n}) = \mathrm{ALED}(\mathrm{norm\_ale\_y}, \mathrm{ale\_n}) \]
NALED produces a score that ranges from 0 to 100%. It is essentially the ALED expressed in percentiles, that is, the average effect of a variable over its full domain of values. So, the NALED of public school status of 5.5 indicates that its average effect on math scores spans the middle 5.5 percent of scores. Academic ratio has an average effect expressed in NALED of 9% of scores.
The median band and random variables
When we do not have p-values, the NALED is particularly helpful in comparing the practical relevance of variables against our threshold for the median band by which we consider that a variable needs to shift the outcome on average by more than 5% of the median values. This threshold is the same scale as the NALED. So, we can tell that public school status with its NALED of 5.5 just barely crosses our threshold.
It is particularly striking to note the ALE effect size measures for
the random rand_norm:
mb_gam_math$ale$stats$by_term$rand_norm
#> # A tibble: 6 × 7
#> statistic estimate p.value conf.low median mean conf.high
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 aled 0.143 0.182 0.00708 0.111 0.143 0.345
#> 2 aler_min -0.734 0.044 -2.59 -0.316 -0.734 -0.0228
#> 3 aler_max 0.444 0.223 0.0243 0.326 0.444 1.13
#> 4 naled 2.23 0.357 0 1.85 2.23 5.09
#> 5 naler_min -8.27 0.142 -27.6 -5.28 -8.27 0
#> 6 naler_max 6.55 0.354 0 5 6.55 15.7rand_norm has a NALED of 2.2. It might be surprising
that a purely random value has any “effect size” to speak of, but
statistically, it must have some numeric value or the other. However, by
setting our default value for the median band at 5%, we effectively
exclude rand_norm from serious consideration. In informal
tests with several different random seeds, the random variables never
exceeded this 5% threshold. Setting the median band too low at a value
like 1% would not have excluded the random variable, but 5% seems like a
nice balance. Thus, the effect of a variable like the discrimination
climate score (discrim, 3.3) should probably not be considered
practically meaningful.
We realize that 5% as a threshold for the median band is rather arbitrary, inspired by traditional \(\alpha\) = 0.05 for statistical significance and confidence intervals. A proper analysis should use p-values, as most of this article does. However, our initial analyses here show that 5% seems to be an effective choice for excluding a purely random variable from consideration, even for quick initial analyses.
We return to using p-values for the rest of this article.
Interpretation of normalized ALE effect sizes
Here we summarize some general principles for interpreting normalized ALE effect sizes.
- Normalized ALE deviation (NALED): this is the
average variation of ALE effect of the input variable.
- Values range from 0 to 100%:
- 0% means no effect at all.
- 100% means the maximum possible effect any variable could have: for a binary variable, one value (50% of the data) sets the outcome at its minimum value and the other value (the other 50% of the data) sets the outcome at its maximum value.
- Larger NALED means stronger effects.
- Values range from 0 to 100%:
- Normalized ALE range (NALER): these are the minimum
and maximum effects of any value of the input variable.
- NALER minimum ranges from –50% to 0%; NALER maximum ranges from 0% to +50%:
- 0% means no effect at all. It indicates that the only effect of the input variable is to keep the outcome at the median of its range of values.
- NALER minimum of n means that, regardless of the effect size of NALED, the minimum effect of any input value shifts the outcome by n percentile points of the outcome range. Lower values (closer to –50%) mean a stronger extreme effect.
- NALER maximum of x means that, regardless of the effect size of NALED, the maximum effect of any input value shifts the outcome by x percentile points of the outcome range. Greater values (closer to +50%) mean a stronger extreme effect.
In general, regardless of the values of ALE statistics, we should always visually inspect the ALE plots to identify and interpret patterns of relationships between inputs and the outcome.
A common question for interpreting effect sizes is, “How strong does an effect need to be to be considered ‘strong’ or ‘weak’?” On one hand, we refuse to offer general guidelines for how “strong” is “strong”. The simple answer is that it depends entirely on the applied context. It is not meaningful to try to propose numerical values for statistics that are supposed to be useful for all applied contexts.
On the other hand, we do consider it very important to delineate the threshold between random effects and non-random effects. It is always important to distinguish between a weak but real effect from one that is just a statistical artifact due to random chance. For that, we can offer some general guidelines based on whether or not we have p-values.
When we have p-values for ALE statistics, then the boundaries of ALER
should generally be used to determine the acceptable risk of considering
a statistic to be meaningful. Statistically significant ALE effects are
those that are less than the 0.05 p-value ALER minimum of a random
variable and greater than the 0.05 p-value maximum of a random variable.
As we explained above when introducing the ALER band, this is precisely
what the {ale} package does, especially in the plots that
highlight the ALER band and the confidence region tables that use the
specified ALER p-value threshold.
In the absence of p-values, we suggest that NALED can be a general guide for non-random values. In our informal tests, we find that NALED values below 5% have the same average effect as a random variable. That is, the average effect is not reliable; it might be random. However, regardless of the average effect indicated by NALED, large NALER effects indicate that the ALE plot should be inspected to interpret the exceptional cases. This caveat is very important; unlike GLM coefficients, ALE analysis is sensitive to exceptions to the overall trend. This is precisely what makes it valuable for detecting non-linear effects.
In general, if NALED < 5%, NALER minimum > –5%, and NALER maximum < +5%, the input variable has no meaningful effect. All other cases are worth inspecting the ALE plots for careful interpretation: - NALED > 5% means a meaningful average effect. - NALER minimum < –5% means that there might be at least one input value that significantly lowers the outcome values. - NALER maximum > +5% means that there might be at least one input value that significantly increases the outcome values.
