The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
misskappa estimates weighted agreement coefficients
(Conger, Fleiss, Brennan–Prediger) when some ratings are
missing, with standard errors, confidence intervals,
and equality tests.
dat.gwet2014 is a small rating matrix (20 subjects, five
raters, four ordered categories) with some ratings missing
(NA). The default estimator, inverse-probability weighting
("ipw"), is consistent when ratings are missing completely
at random (MCAR) and is the recommended starting point:
fit <- agreement(dat.gwet2014, weight = "linear")
fit
#> misskappa: estimator = ipw, weight = linear, n = 20
#> estimate se lower upper
#> Conger 0.5879 0.0772 0.4365 0.7393
#> Fleiss 0.5858 0.0774 0.4341 0.7375
#> Brennan-Prediger 0.6397 0.0767 0.4894 0.7900Ratings can be numbers, strings, or factors.
categories = 0:4 would declare a fifth, unused category,
which changes Brennan–Prediger and the linear weights. The print method
shows each coefficient with its standard error and a 95% confidence
interval. The pieces are available directly:
coef(fit)
#> Conger Fleiss Brennan-Prediger
#> 0.5879222 0.5858274 0.6397286
sqrt(diag(vcov(fit)))
#> Conger Fleiss Brennan-Prediger
#> 0.07724798 0.07739332 0.07668414confint() gives natural-scale Wald intervals, or a
variance-stabilizing Fisher z interval that always stays inside
(-1, 1), useful when a coefficient is close to the
boundary:
confint(fit)
#> 2.5 % 97.5 %
#> Conger 0.4365189 0.7393254
#> Fleiss 0.4341393 0.7375156
#> Brennan-Prediger 0.4894304 0.7900267
#> attr(,"method")
#> [1] "wald"
#> attr(,"transform")
#> [1] "none"
#> attr(,"boundary_fallback")
#> Conger Fleiss Brennan-Prediger
#> FALSE FALSE FALSE
confint(fit, transform = "fisher")
#> 2.5 % 97.5 %
#> Conger 0.4162142 0.7191417
#> Fleiss 0.4139274 0.7173856
#> Brennan-Prediger 0.4647036 0.7666434
#> attr(,"method")
#> [1] "wald"
#> attr(,"transform")
#> [1] "fisher"
#> attr(,"boundary_fallback")
#> Conger Fleiss Brennan-Prediger
#> FALSE FALSE FALSEOn complete data with few subjects,
confint(fit, method = "jackknife") gives exact delete-one
jackknife intervals with a t critical value instead.
?confint.misskappa_estimate lists the fits that support it.
For a tidy summary, as.data.frame(fit) returns one row per
coefficient with its estimate, standard error, and interval.
Start with "ipw". The others suit specific
situations:
| Function and data | estimator |
Assumption |
|---|---|---|
agreement(), categorical ratings |
"ipw" |
MCAR, rater-varying observation rates |
agreement(), categorical ratings |
"cat_fiml" |
Ignorable (MCAR or MAR), saturated multinomial |
agreement(), scored ratings |
"pairwise" |
MCAR, distribution-free, saturated covariance |
agreement(), scored ratings |
"nt_fiml" |
Ignorable, normal-theory, saturated covariance |
agreement(), scored ratings |
"nt_fiml_cs" |
Ignorable, normal-theory, compound-symmetry covariance |
agreement_counts(), category counts |
"pairwise" |
MCAR, exchangeable raters |
agreement_counts(), category counts |
"cat_fiml" |
Ignorable, exchangeable raters |
agreement_counts(), category counts |
"nt_fiml_cs" |
Ignorable, exchangeable raters, normal-theory |
Use "cat_fiml" when missingness may depend on the
observed ratings (MAR) and there are few raters and categories. The
scored estimators treat the categories as numbers and use the quadratic
weight. They report Brennan–Prediger only when the scale is
declared:
agreement(dat.zapf2016, estimator = "pairwise")
#> misskappa: estimator = pairwise, weight = quadratic, n = 50
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.953
#> Fleiss 0.8984 0.0279 0.8437 0.953
agreement(dat.zapf2016, estimator = "nt_fiml", categories = 1:5)
#> misskappa: estimator = nt_fiml, weight = quadratic, n = 50
#> estimate se lower upper
#> Conger 0.8985 0.0278 0.8439 0.9530
#> Fleiss 0.8984 0.0279 0.8437 0.9530
#> Brennan-Prediger 0.8675 0.0313 0.8062 0.9288"nt_fiml_cs" assumes compound symmetry, which lets it
work on planned-incomplete designs where some rater pairs never rate the
same subject. The estimators
article explains the trade-offs and works through such a design.
When raters are interchangeable and you only have per-subject
category counts, use agreement_counts().
dat.fleiss1971 is the classic psychiatric-diagnosis table
(30 subjects, six raters, five categories):
agreement_counts(dat.fleiss1971)
#> misskappa: estimator = pairwise, weight = nominal, n = 30
#> estimate se lower upper
#> Fleiss 0.4302 0.0533 0.3258 0.5347
#> Brennan-Prediger 0.4444 0.0542 0.3382 0.5507ratings_to_counts() turns a subjects-by-raters matrix
into this format.
Set g > 2 for kernels that compare g
raters at once, or pass a subjects-by-raters-by-features array for
vector-valued ratings. The beyond
pairwise agreement article covers both.
Every fit carries a per-subject influence matrix fit$psi
with vcov == crossprod(psi) / n^2. That is what a Wald test
needs, so agreement_test() compares a coefficient across
fits and returns an htest. Only the covariance of the
contrast changes between designs:
fit$psiPass a single fit and a value. This tests whether the
(nominal) Conger coefficient for the Gwet (2014) data equals 0.5. Conger
is the first coefficient of a agreement() fit, so it is the
default coef:
agreement_test(agreement(dat.gwet2014, estimator = "ipw"), value = 0.5)
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
#>
#> One-sample Wald test of Conger
#>
#> data: fit1
#> X-squared = 0.46985, df = 1, p-value = 0.4931
#> alternative hypothesis: true Conger is not equal to 0.5
#> sample estimates:
#> fit1
#> 0.4233051When the fits come from disjoint sets of subjects the variances add.
This is the default (paired = FALSE). Here the Conger
coefficient is compared between the Gwet (2014) and Klein (2018)
studies:
agreement_test(
gwet = agreement(dat.gwet2014, estimator = "ipw"),
klein = agreement(dat.klein2018, estimator = "ipw")
)
#> Warning: rater pair(s) rater4-rater5 co-observed by only one subject; the
#> corresponding pairwise covariance is degenerate and the standard error
#> unreliable.
#>
#> Independent-sample Wald test of equal Conger across 2 fits
#>
#> data: gwet, klein
#> X-squared = 0.0019646, df = 1, p-value = 0.9646
#> alternative hypothesis: true difference in Conger is not equal to 0
#> sample estimates:
#> gwet klein
#> 0.4233051 0.4301049When both coefficients are computed on the same subjects,
the two estimates are correlated and ignoring that dependence gives the
wrong standard error. paired = TRUE takes the dependence
from the influence functions.
A clean illustration on real crowdsourced data: in
dat.mcduff2019, MTurk judges rate smiling images on two
attributes (smile presence and image positivity). Each
image is rated on a 1-6 scale by only a handful of the 121 judges, and
almost no two judges share images, so the judges are treated as
exchangeable and each image is summarised by its category counts.
Because both coefficients are estimated on the same items, the contrast
is dependent.
to_counts <- function(attribute) {
m <- with(dat.mcduff2019,
tapply(get(attribute), list(item, judge), function(z) z[1]))
ratings_to_counts(matrix(as.numeric(m), nrow = nrow(m)))
}
smile <- agreement_counts(to_counts("rating_smile"), weight = "quadratic")
positive <- agreement_counts(to_counts("rating_positive"), weight = "quadratic")
agreement_test(smile = smile, positive = positive, paired = TRUE)
#>
#> Paired Wald test of equal Fleiss across 2 fits
#>
#> data: smile, positive
#> X-squared = 0.013445, df = 1, p-value = 0.9077
#> alternative hypothesis: true difference in Fleiss is not equal to 0
#> sample estimates:
#> smile positive
#> 0.6421397 0.6451303With more than two fits the test becomes the joint hypothesis that
they are all equal, with G - 1 degrees of freedom.
Is the two-rater quadratic coefficient constant across all rater pairs? The Zapf (2016) data has four raters and hence six pairs, all on the same 50 subjects:
pairs <- combn(4, 2)
pair_fits <- apply(pairs, 2, function(rs)
agreement(dat.zapf2016[, rs], estimator = "pairwise"), simplify = FALSE)
names(pair_fits) <- apply(pairs, 2, paste0, collapse = "")
do.call(agreement_test, c(pair_fits, list(paired = TRUE)))
#>
#> Paired Wald test of equal Conger across 6 fits
#>
#> data: 12, 13, 14, 23, 24, 34
#> X-squared = 5.4178, df = 5, p-value = 0.367
#> sample estimates:
#> 12 13 14 23 24 34
#> 0.9309091 0.8811545 0.9012694 0.8786611 0.9403509 0.8466753agreement_test() covers equality of one coefficient. For
anything else, build the test from the influence rows: a linear contrast
of coefficients has influence function equal to the same linear
combination of the columns of psi
(stats::influence(fit) returns the same matrix). Are Conger
and Brennan-Prediger equal within one fit?
fit <- agreement(dat.tenhove2025, estimator = "nt_fiml_cs", categories = 1:5)
psi <- influence(fit)
est <- coef(fit)[["Conger"]] - coef(fit)[["Brennan-Prediger"]]
se <- sqrt(sum((psi[, "Conger"] - psi[, "Brennan-Prediger"])^2)) / nrow(psi)
c(estimate = est, se = se, z = est / se)
#> estimate se z
#> -0.12743311 0.03771282 -3.37903988For fits on the same subjects, subtract their influence columns row by row in the same way; the rows are named by subject.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.