For my master thesis I am comparing Models using SEM (lavaan). I conducted an online survey to validate new scales to assess ict stress factors and resources. Since the questionnaire was quite long and I suspected there might be problems with response styles I used the method of Weijters et al. (2008) to model different response tendencies (e.g. Aquiescence, ARS).
Part of examining construct validity involves analyizing the indicator loadings of the observed variables on the latent construct in the measurement models (MacKenzie et al., 2011) .
I want to check if these factor loadings are different, if I include ARS in the model. What I noticed was, that the standard errors of the factor loadings are twice or thrice as high if I include ARS. Some factor loadings get higher in this model and all load significantly on the latent construct. What I am not certain of, is if the change in the factor loadings is due to the higher standard errors (what I expect) or due to ARS.
How can I compare factor loadings between models that are not nested? Comparing Modelfits does not help me, since this does not inform me if the single parameters differ. (the Models that include ARS almost always fit better, even though the Effect of ARS on the indicators is not significant)
I cant include the data file, but I extracted some of the code and the output of lavaan. The question concerns the changes in the factor loadings (std.all) of the indicators ICTPAss_day1 - ICTPAss_day3
# Latent Variables:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ICTPAss =~
# ICTPAss_day1 0.817 0.061 13.434 0.000 0.817 0.774
# ICTPAss_day3 0.833 0.058 14.410 0.000 0.833 0.779
# ICTPAss_day4 0.732 0.057 12.811 0.000 0.732 0.641
and
# Latent Variables:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ICTPAss =~
# ICTPAss_d1 0.881 0.242 3.642 0.000 0.881 0.832
# ICTPAss_d3 0.875 0.239 3.654 0.000 0.875 0.821
# ICTPAss_d4 0.749 0.226 3.319 0.001 0.749 0.656
whole Code:
model_ictpass <- '
# measurement model
ICTPAss =~ ICTPAss_day1 + ICTPAss_day3 + ICTPAss_day4
'
fit_ictpass <- lavaan::cfa(model_ictpass, my_data_validity, estimator = "MLM", std.lv = T)
summary(fit_ictpass, fit.measures=T, standardized=T, rsquare = T)
# lavaan 0.6-8 ended normally after 13 iterations
#
# Estimator ML
# Optimization method NLMINB
# Number of model parameters 6
#
# Number of observations 356
#
# Model Test User Model:
# Standard Robust
# Test Statistic 0.000 0.000
# Degrees of freedom 0 0
#
# Model Test Baseline Model:
#
# Test statistic 292.653 230.441
# Degrees of freedom 3 3
# P-value 0.000 0.000
# Scaling correction factor 1.270
#
# User Model versus Baseline Model:
#
# Comparative Fit Index (CFI) 1.000 1.000
# Tucker-Lewis Index (TLI) 1.000 1.000
#
# Robust Comparative Fit Index (CFI) NA
# Robust Tucker-Lewis Index (TLI) NA
#
# Loglikelihood and Information Criteria:
#
# Loglikelihood user model (H0) -1459.113 -1459.113
# Loglikelihood unrestricted model (H1) -1459.113 -1459.113
#
# Akaike (AIC) 2930.226 2930.226
# Bayesian (BIC) 2953.475 2953.475
# Sample-size adjusted Bayesian (BIC) 2934.440 2934.440
#
# Root Mean Square Error of Approximation:
#
# RMSEA 0.000 0.000
# 90 Percent confidence interval - lower 0.000 0.000
# 90 Percent confidence interval - upper 0.000 0.000
# P-value RMSEA <= 0.05 NA NA
#
# Robust RMSEA 0.000
# 90 Percent confidence interval - lower 0.000
# 90 Percent confidence interval - upper 0.000
#
# Standardized Root Mean Square Residual:
#
# SRMR 0.000 0.000
#
# Parameter Estimates:
#
# Standard errors Robust.sem
# Information Expected
# Information saturated (h1) model Structured
#
# Latent Variables:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ICTPAss =~
# ICTPAss_day1 0.817 0.061 13.434 0.000 0.817 0.774
# ICTPAss_day3 0.833 0.058 14.410 0.000 0.833 0.779
# ICTPAss_day4 0.732 0.057 12.811 0.000 0.732 0.641
#
# Variances:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# .ICTPAss_day1 0.447 0.074 6.079 0.000 0.447 0.401
# .ICTPAss_day3 0.449 0.072 6.194 0.000 0.449 0.393
# .ICTPAss_day4 0.767 0.075 10.246 0.000 0.767 0.589
# ICTPAss 1.000 1.000 1.000
#
# R-Square:
# Estimate
# ICTPAss_day1 0.599
# ICTPAss_day3 0.607
# ICTPAss_day4 0.411
#
####6.3.0.1 Messmodell ARS
model_ictpass_ars <- '
# measurement model
ICTPAss =~ ICTPAss_day1 + ICTPAss_day3 + ICTPAss_day4
ARS =~ a*ARS_a + a*ARS_b + a*ARS_c
# time-invariant autoregressive effect
ARS_b ~ e*ARS_a
ARS_c ~ e*ARS_b
# Influence of ARS
ICTPAss_day1 ~ l*ARS
ICTPAss_day3 ~ l*ARS
ICTPAss_day4 ~ l*ARS
'
fit_ictpass_ars <- lavaan::cfa(model_ictpass_ars, my_data_validity, estimator = "MLM", std.lv = T)
summary(fit_ictpass_ars, fit.measures=T, standardized=T, rsquare = T)
# lavaan 0.6-8 ended normally after 36 iterations
#
# Estimator ML
# Optimization method NLMINB
# Number of model parameters 18
# Number of equality constraints 5
#
# Number of observations 356
#
# Model Test User Model:
# Standard Robust
# Test Statistic 8.158 7.800
# Degrees of freedom 8 8
# P-value (Chi-square) 0.418 0.453
# Scaling correction factor 1.046
# Satorra-Bentler correction
#
# Model Test Baseline Model:
#
# Test statistic 379.734 341.671
# Degrees of freedom 15 15
# P-value 0.000 0.000
# Scaling correction factor 1.111
#
# User Model versus Baseline Model:
#
# Comparative Fit Index (CFI) 1.000 1.000
# Tucker-Lewis Index (TLI) 0.999 1.001
#
# Robust Comparative Fit Index (CFI) 1.000
# Robust Tucker-Lewis Index (TLI) 1.001
#
# Loglikelihood and Information Criteria:
#
# Loglikelihood user model (H0) -2035.784 -2035.784
# Loglikelihood unrestricted model (H1) -2031.706 -2031.706
#
# Akaike (AIC) 4097.569 4097.569
# Bayesian (BIC) 4147.943 4147.943
# Sample-size adjusted Bayesian (BIC) 4106.701 4106.701
#
# Root Mean Square Error of Approximation:
#
# RMSEA 0.007 0.000
# 90 Percent confidence interval - lower 0.000 0.000
# 90 Percent confidence interval - upper 0.063 0.060
# P-value RMSEA <= 0.05 0.863 0.886
#
# Robust RMSEA 0.000
# 90 Percent confidence interval - lower 0.000
# 90 Percent confidence interval - upper 0.063
#
# Standardized Root Mean Square Residual:
#
# SRMR 0.029 0.029
#
# Parameter Estimates:
#
# Standard errors Robust.sem
# Information Expected
# Information saturated (h1) model Structured
#
# Latent Variables:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ICTPAss =~
# ICTPAss_d1 0.881 0.242 3.642 0.000 0.881 0.832
# ICTPAss_d3 0.875 0.239 3.654 0.000 0.875 0.821
# ICTPAss_d4 0.749 0.226 3.319 0.001 0.749 0.656
# ARS =~
# ARS_a (a) 0.207 0.024 8.679 0.000 0.207 0.554
# ARS_b (a) 0.207 0.024 8.679 0.000 0.207 0.433
# ARS_c (a) 0.207 0.024 8.679 0.000 0.207 0.465
#
# Regressions:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ARS_b ~
# ARS_a (e) 0.028 0.059 0.478 0.633 0.028 0.022
# ARS_c ~
# ARS_b (e) 0.028 0.059 0.478 0.633 0.028 0.030
# ICTPAss_day1 ~
# ARS (l) 0.642 0.492 1.305 0.192 0.642 0.607
# ICTPAss_day3 ~
# ARS (l) 0.642 0.492 1.305 0.192 0.642 0.603
# ICTPAss_day4 ~
# ARS (l) 0.642 0.492 1.305 0.192 0.642 0.563
#
# Covariances:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# ICTPAss ~~
# ARS -0.446 0.475 -0.940 0.347 -0.446 -0.446
#
# Variances:
# Estimate Std.Err z-value P(>|z|) Std.lv Std.all
# .ICTPAss_day1 0.438 0.073 6.023 0.000 0.438 0.391
# .ICTPAss_day3 0.458 0.069 6.620 0.000 0.458 0.404
# .ICTPAss_day4 0.759 0.071 10.666 0.000 0.759 0.582
# .ARS_a 0.097 0.011 8.690 0.000 0.097 0.693
# .ARS_b 0.183 0.016 11.605 0.000 0.183 0.801
# .ARS_c 0.154 0.015 9.923 0.000 0.154 0.771
# ICTPAss 1.000 1.000 1.000
# ARS 1.000 1.000 1.000
#
# R-Square:
# Estimate
# ICTPAss_day1 0.609
# ICTPAss_day3 0.596
# ICTPAss_day4 0.418
# ARS_a 0.307
# ARS_b 0.199
# ARS_c 0.229
Literature:
MacKenzie, S. B., Podsakoff, P. M. & Podsakoff, N. P. (2011). Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and Existing Techniques. MIS Quarterly, 35(2), 293. https://doi.org/10.2307/23044045
Weijters, B., Schillewaert, N. & Geuens, M. (2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36(3), 409–422. https://doi.org/10.1007/s11747-007-0077-6
Weijters, B., Geuens, M. & Schillewaert, N. (2010a). The Individual Consistency of Acquiescence and Extreme Response Style in Self-Report Questionnaires. Applied Psychological Measurement, 34(2), 105–121. https://doi.org/10.1177/0146621609338593
Thanks for any answer and greetings :)