Example Workflow
This section walks through the CALMs application interface using screenshots for illustrative purposes. Specifically, we analyze data from the 2015 Work Orientations Survey that includes responses from the United States (USA) and the United Kingdom (UK; ISSP Research Group, 2017). The 2015 Work Orientations dataset is from an international project that began in 1984 and was collected across 37 countries (ISSP Research Group, 2017).
The portion of the 2015 Work Orientations dataset used for the demonstration includes 1,477 responses from the USA and 1,793 responses from the UK. We specifically chose the stated two countries because full scalar invariance was not supported in previous measurement invariance studies using the constructs quality of job context (JX), quality of job content (JC), and quality of work environment (WE) in the measurement model using the 1989 Work Orientations dataset (Cheung & Lau, 2012; Cheung & Rensvold, 1999).
The 2015 Work Orientations dataset provided data for two of these previously utilized constructs, JC and WE (ISSP Research Group, 2017). Each construct is measured by three items, scored on a five-point Likert-type scale ranging from 1 (strongly agree) to 5 (strongly disagree). Figure 1 depicts the 2-factor measurement model used in the illustrative example. What follows next is a recommend set of steps to comprehensively analyze the latent means of JC and WE by country, where country is either USA or UK. Note that researchers may choose to use the application in a different way that the example workflow and skip tests if that fits their research scenario.
Figure 1. Measurement Model
Step 1: Load Data
Users can either use the built-in dataset by leaving Use 2015 Work Orientations Survey Data selected, or upload their own data by deselecting this option.
To upload your own dataset and accompanying *Meta.csv file, follow the steps shown in the GIF below.
Step 2: View Data
The labeling of the items in the original dataset (ISSP Research Group, 2017) was not intuitive for our illustrative example; hence, the original items were renamed as previously described and as depicted in Figure 2.
Figure 2. View Data Tab
Step 3: Check Group Equivalency
CALMs uses the MatchIt package in R (Ho et al., 2011) for propensity score analyis including checking for group equivalency. The comparison groups for the demonstration with the 2015 Work Orientations Survey data are the USA and the UK. Hence, USA was selected as the Grouping Variable. All possible covariates were selected as Covariates to Check.
Figure 3. Check Group Equivalency Tab
The output in Figure 3 shows that there are statistically (p < .05) and practically significant (Cramer’s V > .10) differences in employment type and degree by country. Specifically, employment type (SelfEmp) was found to be statistically significant while organization type (PrivateOrg) was found to be both statistically and practically significant.
Step 4: Propensity Score Analysis Setup
CALMs offers users the flexibility to use either a default call to MatchIt or to define a custom call. A link to the matchIt documentation is included within the application.
To customize the call, deselect Use Default call to matchit and edit the arugments following data=dpsm in the provided code box.
Two propensity score matching (PSM) methods, nearest neighbor and genetic matching, are the most common.
Nearest neighbor matching requires the input of all demographic variables and has been recommended as the most straightforward PSM method (Caliendo & Kopeinig, 2008; Keiffer & Lane, 2016). Although nearest neighbor is computationally effective with large datasets, the method pairs each treated unit with its nearest matching control without considering fully optimizing matches. This may lead to covariates not being optimally balanced with less equivalent groups, as compared to more stringent or robust matching methods.
Genetic matching is recommended when PSM output is required to have highly equivalent groups as it effectively achieves good matching balance even with highly complex data (Randolph et al., 2014). Genetic matching requires the input of all demographic variables (e.g., gender, age group, race/ethnicity, and educational level) into an algorithm that results in statistically (e.g., p ≤ .05) and practically (e.g., Cramer’s V ≥ .10) significant differences.
By default, CALMs uses the nearest neighbor method.
Figure 4. Propensity Score Analysis Setup Tab
Step 5: Propensity Score Analysis
Figure 5 presents the result of the propensity score analysis using the default call previously described.
Figure 5. Propensity Score Analysis Results Tab
The nearest neighbor method yielded two equivalent groups with 769 responses in each country. We observed that there were statistically significant differences in gender (Male) by country and elected to use the results of the nearest neighbor method since there was not a practically significant difference by country (all Cramer’s V < .10).
Step 6: Measuremenent Invariance Tests
When conducting measurement invariance tests, the application defaults to using the matched dataset.
To change this, users can deselect Use matched data for invariance tests.
Users can also select the Grouping Variable and
Items to Analyze. By default, the application include
all items identified in the *Meta.csv file as type
item.
Measurement invariance tests include configural, metric, and scalar. Omnibus and scale-level tests are provided for both metric and scale invariance tests. Commonly recommended fit indices criteria include: (a) comparative fit index (CFI) ≥ .95; (b) standardized root-mean-square residuals (SRMR) ≤ .05; and (c) root-mean-squared-error of approximation (RMSEA) .05 to .08 (Kline, 2016; Schumacker & Lomax, 2016).
Statistically significant model noninvariance is determined based on the p-value of the χ² difference test at p ≤ .05 (Cheung & Rensvold, 1999; van de Schoot et al., 2012). Guidelines have been provided to evaluate the ΔCFI for practical model (non)invariance, namely: (a) practical model invariance for ΔCFI ≥ -.01; (b) potential practical model noninvariance for ΔCFI between -.01 and -.02; and (c) practical model noninvariance for ΔCFI ≤ -.02 (Cheung & Rensvold, 2002).
Figure 6. Measurement Invariance Tab
The results of measurement invariance tests (see Figure 6) indicated that the configural model showed good fit. This model indicated good fit with a SRMR = .031 and CFI = .956. The metric model was compared to the configural model and met criteria for both statistical and practical invariance (Δχ²[4] = 8.484, p = .075; ΔCFI = -.004). However, the data did not reach the thresholds for scalar invariance (Δχ²[4] = 55.772, p < .001; ΔCFI = -.042). Both the JC (Δχ²[2] = 24.996, p < .001; = -.019) and WE (Δχ²[2] = 30.796, p < .001; ΔCFI = -.023) scales demonstrated evidence of scalar non-invariance.
Step 7: Metric Invariance Tests
In our illustrative example, it was not necessary to conduct follow-up tests for metric invariance as neither the omnibus test nor the scale-level tests for JC and WE indicated evidence of metric non-invariance.
However, for demonstration purposes, we conducted metric invariance tests specifically on the JC scale.
Note that the application uses the p-value of the χ² difference test when determining invariant subsets of items (Cheung & Rensvold, 1999). The default significance level (alpha) is set to .05, but users may adjust this threshold as needed. In this example, we set the alpha to .01 (see Figure 7).
Figure 7. Metric Invariance Tab
The factor ratio test (see Figure 7) confirmed that all JC items were metric invariant. Similarly, all WE items were metric invariant (tests not shown).
Step 8: Scalar Invariance Tests
Because full scalar invariance was not demonstrated, we conducted partial MI testing on each scale. Had we determined that the factor loadings were non-invariant at the metric invariance assessment, we could have allowed a set of loadings to be freely estimated to allow for a partial scalar invariance assessment.
Figure 8. Scalar Invariance Tab
The factor ratio test (see Figure 8) identified JC2 and JC3 as a invariant subset of JC items (p > .01). Similarly, WE1 and WE2 were identified (tests not shown) as a invariant subset of WE items (p > .01).
Based on the results of the scalar invariance assessment, the intercepts for WE3 and JC1 should be freely estimated to account for the partial scalar invariance.
Step 9: Structural Invariance Tests
Building on the results of the scalar invariance testing, we allowed the intercepts for WE3 and JC1 to be freely estimated. Structural invariance is given when the comparison between an unconstrained and a constrained structural model yields a non-significant χ² difference (p > .05) and a non-significant CFI difference (Cheung & Rensvold, 1999; Cheung & Rensvold, 2002; Kline, 2016; Schumacker & Lomax, 2016).
Figure 9. Structural Invariance Tab
The results indicate that the set of scales met the criteria for structural invariance. Although the structural invariance model was statistically significantly different from the scalar model (Δχ²[2] = 6.374, p = .041), the difference was not practically significant (ΔCFI = -.004; see Figure 9). However, considering only JC, a statistically significant latent mean difference was observed (-.077, p = .013). Given that the latent mean for the USA was constrained to zero, the negative estimate indicates that the latent mean for the UK is lower in JC. There was no significant latent mean difference for WE across the two countries (-.047, p = .326).