Dispersion scores may exceed minimum or maximum
Another issue arises when we are dealing with corpus parts of (very)
different length. Distributions that would then be considered as
minimally/maximally pervasive or minimally/maximally even may yield a
dispersion score that is less extreme than that based on the actual
distribution of subfrequencies. Whether a particular measure is affected
by this depends on how the dispersion extremes were constructed,
i.e. whether they were composed to minimize/maximize evenness or
pervasiveness:
- For S, DP, and DKL,
which are evenness measures, dispersion extremes that are defined based
on evenness work well.
- For Range, which is a pervasiveness measure, dispersion
extremes that are defined based on pervasiveness work
well.
- Frequency adjustment for D, D2, and
DA, on the other hand, may yield dispersion scores
outside the [0,1] interval for both methods, i.e. when dispersion
extremes are defined based on evenness or pervasiveness.
This translated into the following preliminary recommendations:
- If corpus parts are fairly similar in size, no problems should
arise. It should also matter little whether frequency adjustment is
based on pervasiveness or evenness extremes, as both schemes then
produce very similar distributions.
- If corpus parts vary considerably in size, meaning that there are
some very long parts, and several very short ones, frequency adjustment
should proceed with caution. Unfortunately, specific criteria cannot be
given at the moment. In such situations, the safest strategy seems to be
the following:
- Do not calculate frequency-adjusted scores for D,
D2, and DA.
- For S, DP, and DKL,
use
freq_adjust_method = "even".
- For Range, use
freq_adjust_method = "pervasive".
For S, DP, DKL, and
Range, these recommendations are implemented as defaults in the
corresponding functions
(i.e. disp_S()/disp_S_tdm(),
disp_DP()/disp_DP_tdm(),
disp_DKL()/disp_DKL_tdm(),
disp_R()/disp_R_tdm()).
While more work is required to understand how dispersion measures
respond to different dispersion extremes, some observations may be made.
Thus, it easy to see why Range will be affected if dispersion
extremes are defined based on evenness: For a minimally even
distribution, we fill in corpus parts starting with the smallest one(s).
The number of parts containing the item may therefore exceed that in the
actual corpus – especially if the smallest parts are very small.
The reason why S, DP, and
DKL do not work well with maximally/minimally
pervasive distributions could be due to the fact that the size of corpus
parts is completely disregarded, which can lead to very disproportionate
distributions of subfrequencies. Thus, if we allocate one occurrence to
each corpus part, this may produce an exceptionally high normalized
frequency for the shortest part, but a very low one for the longest
part.
It is less straightforward to see why D,
D2, and DA may perform poorly
under both schemes. What these measures have in common, however, is that
they rely on normalized subfrequencies. Since the allocation scheme
fills in corpus parts until they are “full” (see issue 1 above),
extremely high normalized frequencies may result. This is especially
problematic if the smallest corpus parts are very small.
It seems that Biber et al.’s (2016) set of lexical items is
particularly suitable for further investigation of these issues. Coupled
with a set of corpus parts that differ dramatically in size, we can
identify which measures and items are affected.
The built-in data set biber150_spokenBNC1994 considers
individual speakers in the Spoken BNC1994 as corpus parts. The word
counts are distributed very unevenly across speakers; they range from 1
to 68,136. The extremely skewed distribution of part sizes is apparent
in the following histogram:
oldpar <- par(mar = c(5.1, 4.1, 4.1, 2.1))
par(mar = c(4, 4, 1, 0.3), xpd = TRUE)
hist(
biber150_spokenBNC1994[1,],
main = NULL,
xlab = "Size of corpus parts (speakers)",
breaks = seq(0, 70000, length=40),
col = "grey60")
We start by calculating frequency-adjusted scores for the 150 items
in this dataset. Note that we use unit_interval = FALSE to
avoid forcing adjusted scores into the unit interval (i.e. replacing
scores smaller than 0 with 0, and scores greater than 1 with 1).
DM_even <- disp_tdm(
biber150_spokenBNC1994,
row_partsize = "first",
freq_adjust = TRUE,
freq_adjust_method = "even",
unit_interval = FALSE,
print_score = FALSE,
verbose = FALSE,
suppress_warning = TRUE)
DM_pervasive <- disp_tdm(
biber150_spokenBNC1994,
row_partsize = "first",
freq_adjust = TRUE,
freq_adjust_method = "pervasive",
unit_interval = FALSE,
print_score = FALSE,
verbose = FALSE,
suppress_warning = TRUE)
We then examine, for each dispersion measure, the range of scores
across the 150 items, starting with the adjusted scores based on
freq_adjust_method = "even":
round(
apply(
DM_even,
2,
range, na.rm = TRUE),
2)
#> Rrel_nofreq D_nofreq D2_nofreq S_nofreq DP_nofreq DA_nofreq DKL_nofreq
#> [1,] -0.07 -4.35 -1.89 0.00 0.00 -0.77 0.10
#> [2,] 1.00 0.63 0.85 0.99 0.93 0.60 0.96
Then we look at how many items are affected by this, for each
measure:
apply(
DM_even,
2,
function(x){
sum(x < 0 | x > 1, na.rm = TRUE)
})
#> Rrel_nofreq D_nofreq D2_nofreq S_nofreq DP_nofreq DA_nofreq
#> 3 67 15 0 0 19
#> DKL_nofreq
#> 0
Next, we consider the range of scores for adjusted based on
freq_adjust_method = "pervasive":
round(
apply(
DM_pervasive,
2,
range, na.rm = TRUE),
2)
#> Rrel_nofreq D_nofreq D2_nofreq S_nofreq DP_nofreq DA_nofreq DKL_nofreq
#> [1,] 0 0.00 0.00 -0.67 -0.53 0.00 -1.29
#> [2,] 1 1.06 1.16 1.76 2.05 2.49 4.06
Number of items affected by this, for each measure:
apply(
DM_pervasive,
2,
function(x){
sum(x < 0 | x > 1, na.rm = TRUE)
})
#> Rrel_nofreq D_nofreq D2_nofreq S_nofreq DP_nofreq DA_nofreq
#> 0 60 61 75 88 62
#> DKL_nofreq
#> 92
