The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
Overview
This vignette introduces the package’s three recommended
content-pretesting workflows—item sorting,
construct ratings, and expert panels.
sort_validity(), rating_validity(), and
expert_validity() organize quantitative evidence while
keeping substantive decisions separate from statistical flags.
A common workflow contract
All three fitted workflow objects expose results,
scale_summary, settings, design,
and details. Their result tables also contain a common
status field: Supported, Review,
Insufficient data, or Descriptive only.
Method-specific recommendation wording is preserved alongside that
common status. This makes it possible to write reusable code across
workflows without pretending that a Howard-Melloy retention decision,
Hinkin-Tracey screening result, and expert-panel judgment are
substantively identical.
Sort-based (Psa, Csv, binomial)
toy_sort <- data.frame(
item = rep(paste0("I", 1:4), each = 12),
rater = rep(1:12, 4),
target_construct = rep(c("A","A","B","B"), each = 12),
assigned_construct = c(
sample(c("A","B"), 12, TRUE, c(.80,.20)),
sample(c("A","B"), 12, TRUE, c(.65,.35)),
sample(c("A","B"), 12, TRUE, c(.70,.30)),
sample(c("A","B"), 12, TRUE, c(.45,.55))
)
)
psa <- compute_psa(toy_sort)
csv <- compute_csv(toy_sort)
csv$decision <- vapply(seq_len(nrow(csv)), function(i) {
csv_binom_test(csv$n_target[i], csv$n[i])$decision
}, character(1))
psa; csv
#> item target n_total n n_missing n_target psa psa_low psa_high
#> 1 I1 A 12 12 0 9 0.7500000 0.46769467 0.9110583
#> 2 I2 A 12 12 0 5 0.4166667 0.19326031 0.6804887
#> 3 I3 B 12 12 0 2 0.1666667 0.04696514 0.4480309
#> 4 I4 B 12 12 0 5 0.4166667 0.19326031 0.6804887
#> item target n_total n n_missing n_target competitor n_other_max csv
#> 1 I1 A 12 12 0 9 B 3 0.5000000
#> 2 I2 A 12 12 0 5 B 7 -0.1666667
#> 3 I3 B 12 12 0 2 A 10 -0.6666667
#> 4 I4 B 12 12 0 5 A 7 -0.1666667
#> decision
#> 1 n.s.
#> 2 n.s.
#> 3 n.s.
#> 4 n.s.Interpretation - Psa = share assigning the intended construct. - Csv = margin of wins: $ $. - Binomial test (H0: $ p ≤ .5 $) flags items with above-chance targeting.
Construct-rating workflow (HTC, HTD, repeated-measures ANOVA)
set.seed(2)
toy_ratings <- expand.grid(
item = c("I1", "I2", "I3"),
rater = 1:16,
construct = c("A", "B", "C")
)
toy_ratings$target_construct <- ifelse(toy_ratings$item == "I3", "B", "A")
toy_ratings$rating <- ifelse(
toy_ratings$construct == toy_ratings$target_construct,
pmin(5, pmax(1, round(rnorm(nrow(toy_ratings), 4.4, .6)))),
pmin(5, pmax(1, round(rnorm(nrow(toy_ratings), 2.2, .7))))
)
rating_fit <- rating_validity(toy_ratings, scale_min = 1, scale_max = 5)
rating_fit
#> contentvalidR construct-rating analysis
#> ---------------------------------------
#> Items: 3 | Raters: 16 | Target scales: 2 | Constructs: 3
#> Design: within-judge ratings | Scale: 1 to 5
#> Item inference: one-way repeated-measures ANOVA (Greenhouse-Geisser corrected omnibus p) plus planned paired target-versus-orbiting contrasts
#> Planned-contrast adjustment: none
#> Judges: naive
#>
#> 3 item(s) meet the full item-level screening criterion; 0 item(s) are flagged for review.
#>
#> Item-level evidence:
#> item target n_complete strongest_competitor htc htd p_value max_contrast_p
#> I1 A 16 C 0.825 0.492 0 0
#> I2 A 16 C 0.887 0.508 0 0
#> I3 B 16 C 0.875 0.539 0 0
#> recommendation
#> Retain
#> Retain
#> Retain
#>
#> Target-scale Colquitt benchmark summary:
#> target n_items n_htc n_htd mean_htc htc_strength mean_htd htd_strength
#> A 2 2 2 0.856 Moderate 0.500 Very Strong
#> B 1 1 1 0.875 Strong 0.539 Very Strong
#> benchmark_set
#> overall
#> overall
#>
#> Colquitt labels are empirical percentile norms for scale-level HTC/HTD averages, not universal cutoffs.
#> HTC is an average rating and HTD is a difference between ratings, so they sit on
#> different scales with different typical values. A high HTC can be labeled Weak in
#> the same analysis where a much smaller HTD is labeled Very Strong. Compare each
#> index against its own benchmark, never against the other index's number.
#>
#> What these columns mean
#> htc -- Hinkin-Tracey Correspondence. Average rating of the item against
#> its intended construct definition, expressed as a proportion of the
#> rating scale. (0 to 1; higher is stronger)
#> htd -- Hinkin-Tracey Distinctiveness. How far the intended construct's
#> average rating exceeds the best competing construct's, as a
#> proportion of the rating scale. It is a difference, so its typical
#> values are far smaller than HTC's. (usually a small positive number;
#> higher is stronger)
#>
#> What the status labels mean
#> Supported -- The evidence met the criteria set for this analysis.
#> Review -- Something here needs a closer look. This is not an instruction
#> to delete anything.
#> Insufficient data -- Too little usable data to reach a judgment.
#> Descriptive only -- Reported for description only; no decision rule was
#> applied.
#> Each workflow also uses its own wording in the recommendation column
#> (Retain, Strong support, Typical, Covered, and so on). Those words map
#> onto the shared statuses above.
#>
#> See `contentvalid_glossary()` for all terms, or set
#> `options(contentvalidR.show_key = FALSE)` to hide this key.
#>
#> 'Review' is not an automatic deletion decision. Consider construct definitions, item wording,
#> orbiting-construct choice, domain coverage, and qualitative judge feedback.
summary(rating_fit)
#> Summary of construct-rating content-validity evidence
#> ---------------------------------------------------
#> Retain: 3 of 3 item(s)
#> Review: 0 of 3 item(s)
#>
#> Target-scale evidence:
#> target n_items n_htc n_htd n_retain n_review mean_htc htc_strength mean_htd
#> A 2 2 2 2 0 0.856 Moderate 0.500
#> B 1 1 1 1 0 0.875 Strong 0.539
#> htd_strength overall_strength
#> Very Strong Moderate
#> Very Strong Strong
#>
#> A: Generally supportive normative standing, with at least one content-validity dimension in the moderate range; inspect weaker items and construct overlap before finalizing the scale.
#> B: Strong normative standing on the weaker of definitional correspondence (HTC) and distinctiveness (HTD).
#>
#> All analyzed items met the item-level inferential screening criterion.
#>
#> Interpret these results alongside theory, domain coverage, and qualitative feedback.
#> The analysis does not by itself establish comprehensiveness or the full content-validity argument.Interpretation - HTC summarizes definitional correspondence with the intended construct. - HTD summarizes distinctiveness from orbiting constructs. - The repeated-measures ANOVA tests whether construct-definition ratings differ for an item. - Planned paired contrasts ask the direct screening question: is the target rating significantly higher than every orbiting rating? - Scale-level HTC/HTD averages can be interpreted using Colquitt et al. (2019) empirical norms when the judge population matches their intended use.
Expert-panel workflow
expert_ratings <- matrix(
c(4,4,4,4,4,4,
4,4,4,3,4,4,
4,3,4,4,3,4),
nrow = 6,
dimnames = list(NULL, paste0("Item", 1:3))
)
expert_fit <- expert_validity(expert_ratings, mode = "relevance", lo = 1, hi = 4)
expert_fit
#> contentvalidR expert-panel analysis
#> -----------------------------------
#> Mode: relevance
#> Items: 3 | Experts/item: 6
#> Mean Aiken V: 0.944 | S-CVI/Ave: 1 | S-CVI/UA: 1
#> Strong support: 3 | Support: 0 | Review: 0
#> Panel agreement, Krippendorff's alpha (ordinal): 0.018 (95% interval -0.133
#> to 0.15). Identical rating pairs: 71.1%
#>
#> item N V ci_low ci_high I_CVI I_CVI_low I_CVI_high kappa_mod
#> Item1 6 1.000 0.824 1.000 1 0.61 1 1
#> Item2 6 0.944 0.742 0.990 1 0.61 1 1
#> Item3 6 0.889 0.672 0.969 1 0.61 1 1
#> recommendation
#> Strong support
#> Strong support
#> Strong support
#>
#> ci_low and ci_high bound Aiken's V (Penfield-Giacobbi score interval);
#> I_CVI_low and I_CVI_high bound I-CVI.
#> 95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#> compared seven methods and recommends score intervals over the Wald
#> interval. An interval reflects how few ratings an item received, not
#> whether the right judges were chosen.
#>
#> Panel agreement is one coefficient for the whole panel, whereas kappa_mod
#> describes each item. Alpha can be low when nearly every rating is the same
#> value, even on a panel that agrees closely, so read it beside the share of
#> identical rating pairs. A low alpha with many identical pairs is not by
#> itself evidence of a poor panel. Print `details$agreement` for the full
#> explanation and interval details.
#>
#> CVI thresholds shown by the workflow are common panel-size guidelines, not universal validity cutoffs.
#>
#> What these columns mean
#> V -- Aiken's V. Relevance index that rescales the experts' average rating
#> to run from 0 to 1 given the bounds of the rating scale used. (0 to
#> 1; higher is stronger)
#> I_CVI -- Item-level Content Validity Index. Proportion of experts who
#> rated the item as relevant, after applying the relevance cut. (0 to
#> 1; compared against a panel-size guideline)
#> I_CVI_low/I_CVI_high -- Interval for I-CVI. Lower and upper limits of an
#> interval around I-CVI. Expert panels are usually small, so these
#> intervals are often wide: a single I-CVI value can look more settled
#> than the number of experts behind it supports. (between 0 and 1; the
#> method and level are named in the output)
#> kappa_mod -- Modified kappa. I-CVI adjusted for the chance that experts
#> would have agreed even if rating at random. With small panels, chance
#> agreement is substantial, which is why the raw I-CVI alone can
#> overstate consensus. (0 to 1; higher is stronger)
#> agreement -- Panel-level agreement. One coefficient describing how
#> consistently the whole panel rated the item set: Krippendorff's alpha
#> by default, or Gwet's AC1 if chosen. It is separate from modified
#> kappa, which describes one item at a time. (1 is perfect agreement
#> and 0 is agreement no better than chance; it can be low on a
#> close-agreeing panel whose ratings cluster on one value)
#>
#> What the status labels mean
#> Supported -- The evidence met the criteria set for this analysis.
#> Review -- Something here needs a closer look. This is not an instruction
#> to delete anything.
#> Insufficient data -- Too little usable data to reach a judgment.
#> Descriptive only -- Reported for description only; no decision rule was
#> applied.
#> Each workflow also uses its own wording in the recommendation column
#> (Retain, Strong support, Typical, Covered, and so on). Those words map
#> onto the shared statuses above.
#>
#> See `contentvalid_glossary()` for all terms, or set
#> `options(contentvalidR.show_key = FALSE)` to hide this key.
#>
#> Use quantitative indices alongside expert comments, construct coverage, and comprehensibility review.
summary(expert_fit)
#> Summary of expert-panel content-validity evidence
#> ---------------------------------------------
#> Mode: relevance
#> Supported: 3 | Review: 0
#> Panel agreement, Krippendorff's alpha (ordinal): 0.018 (95% interval -0.133
#> to 0.15). Identical rating pairs: 71.1%
#> No items were flagged by the workflow's quantitative review rules.
#>
#> These summaries support, but do not replace, qualitative content review.Relevance, essentiality, and congruence are intentionally separate
expert tasks. Use mode = "relevance" for Aiken V +
CVI/modified kappa, mode = "essentiality" for Lawshe CVR,
and mode = "congruence" for IOC.
Bundled reproducible examples
The package also installs deterministic CSV examples for the three
workflow families and all expert-panel modes. They are synthetic,
contain no participant data, and are regenerated from
data-raw/build-example-data.R in the source repository.
example_files <- c(
"sort_example.csv",
"rating_example.csv",
"expert_relevance_example.csv",
"expert_essentiality_example.csv",
"expert_congruence_example.csv"
)
vapply(example_files, function(x) {
system.file("extdata", x, package = "contentvalidR")
}, character(1))
#> sort_example.csv
#> "C:/Users/TheGr/AppData/Local/Temp/RtmpgBwMIv/Rinst2ef863d176e5/contentvalidR/extdata/sort_example.csv"
#> rating_example.csv
#> "C:/Users/TheGr/AppData/Local/Temp/RtmpgBwMIv/Rinst2ef863d176e5/contentvalidR/extdata/rating_example.csv"
#> expert_relevance_example.csv
#> "C:/Users/TheGr/AppData/Local/Temp/RtmpgBwMIv/Rinst2ef863d176e5/contentvalidR/extdata/expert_relevance_example.csv"
#> expert_essentiality_example.csv
#> "C:/Users/TheGr/AppData/Local/Temp/RtmpgBwMIv/Rinst2ef863d176e5/contentvalidR/extdata/expert_essentiality_example.csv"
#> expert_congruence_example.csv
#> "C:/Users/TheGr/AppData/Local/Temp/RtmpgBwMIv/Rinst2ef863d176e5/contentvalidR/extdata/expert_congruence_example.csv"See
vignette("reporting-examples", package = "contentvalidR")
for manuscript-ready reporting scaffolds built from those same
files.
Classic indices
R <- matrix(sample(1:5, 5*6, replace = TRUE), nrow = 5)
aikens_v(R, lo = 1, hi = 5)
#> item N n_missing V ci_low ci_high ci_method
#> 1 Item1 5 0 0.60 0.3865815 0.7811935 Penfield-Giacobbi score
#> 2 Item2 5 0 0.30 0.1454772 0.5189728 Penfield-Giacobbi score
#> 3 Item3 5 0 0.70 0.4810272 0.8545228 Penfield-Giacobbi score
#> 4 Item4 5 0 0.35 0.1811918 0.5671457 Penfield-Giacobbi score
#> 5 Item5 5 0 0.60 0.3865815 0.7811935 Penfield-Giacobbi score
#> 6 Item6 5 0 0.65 0.4328543 0.8188082 Penfield-Giacobbi score
cvr(essential = c(8,10,5), N = 12)
#> item ne N cvr p_value critical_ne critical_cvr pass
#> 1 Item1 8 12 0.3333333 0.19384766 10 0.6666667 FALSE
#> 2 Item2 10 12 0.6666667 0.01928711 10 0.6666667 TRUE
#> 3 Item3 5 12 -0.1666667 0.80615234 10 0.6666667 FALSE
M <- matrix(sample(0:1, 6*5, replace = TRUE, prob = c(.3,.7)), nrow = 6)
cvi(M)
#> Content Validity Index (CVI)
#> ----------------------------
#> Items analyzed: 5
#> Judges per item: 6
#> S-CVI/Ave: 0.667
#> S-CVI/UA : 0.200
#>
#> Item-level results (modified kappa is chance-corrected):
#> item A N I_CVI I_CVI_low I_CVI_high Pc kappa_mod
#> Item1 3 6 0.500 0.188 0.812 0.312 0.273
#> Item2 4 6 0.667 0.300 0.903 0.234 0.565
#> Item3 3 6 0.500 0.188 0.812 0.312 0.273
#> Item4 6 6 1.000 0.610 1.000 0.016 1.000
#> Item5 4 6 0.667 0.300 0.903 0.234 0.565
#>
#> 95% intervals for proportions: Wilson score (the default). Newcombe (1998)
#> compared seven methods and recommends score intervals over the Wald
#> interval. An interval reflects how few ratings an item received, not
#> whether the right judges were chosen.
#>
#> Interpretation should consider panel size, item purpose, and qualitative expert feedback;
#> CVI statistics alone do not establish comprehensive content validity.
ioc_df <- data.frame(
item = rep(paste0("I",1:2), each = 9),
judge = rep(1:3, times = 6),
objective = rep(rep(LETTERS[1:3], each = 3), times = 2),
score = sample(c(-1,0,1), 18, replace = TRUE)
)
ioc(ioc_df)
#> item objective n_total n_judges n_missing ioc
#> 1 I1 A 3 3 0 0.0000000
#> 2 I1 B 3 3 0 0.3333333
#> 3 I1 C 3 3 0 -0.6666667
#> 4 I2 A 3 3 0 0.0000000
#> 5 I2 B 3 3 0 0.0000000
#> 6 I2 C 3 3 0 0.0000000Diagnostics & reproducibility
truth <- c(TRUE, TRUE, TRUE, FALSE) # pretend "kept" after CFA
signal_detection(csv$decision == "significant", truth)
#> $confusion
#> Actual
#> Predicted Retain Not retained
#> Retain 0 0
#> Not retained 3 1
#>
#> $accuracy
#> [1] 0.25
#>
#> $sensitivity
#> [1] 0
#>
#> $specificity
#> [1] 1
#>
#> $phi
#> [1] NA
#>
#> $chisq
#> [1] NA
#>
#> $p
#> [1] NA
csv2_sig <- sample(c(TRUE, FALSE), nrow(csv), replace = TRUE)
reproducibility_phi(csv$decision == "significant", csv2_sig)
#> $table
#> Pretest2
#> Pretest1 Retain Not retained
#> Retain 0 0
#> Not retained 1 3
#>
#> $phi
#> [1] NA
#>
#> $chisq
#> [1] NA
#>
#> $p
#> [1] NAPower quick-checks
sort_power(N = c(20, 30), true_p = c(.65, .75))
#> Exact item-sort planning analysis
#> ---------------------------------
#> Retention rule: p0 = 0.50, alpha = 0.050
#>
#> N true_p critical_n_target minimum_observed_psa power
#> 20 0.65 15 0.750 0.245
#> 30 0.65 20 0.667 0.508
#> 20 0.75 15 0.750 0.617
#> 30 0.75 20 0.667 0.894
#>
#> Power is the exact probability of reaching the required target-assignment count
#> under the assumed true target-assignment probability.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.