Estimate Reliability in R with Alpha, Omega, and Kappa | สรุปเนื้อหาที่เกี่ยวข้องv omega rที่มีรายละเอียดมากที่สุด

6 thoughts on “Estimate Reliability in R with Alpha, Omega, and Kappa | สรุปเนื้อหาที่เกี่ยวข้องv omega rที่มีรายละเอียดมากที่สุด”

1. 

   Han says:

   Hello! How do I calculate the test-retest reliability using ICC?

   17/06/2022 at 12:58 PM

   ตอบกลับ
2. 

   Aj says:

   Is there a way to interpret Omega (like we do chronbach's alpha)?

   17/06/2022 at 12:58 PM

   ตอบกลับ
3. 

   Quique Clari says:

   Great video, thank you very much!

   17/06/2022 at 12:58 PM

   ตอบกลับ
4. 

   Mahdi Mazidi says:

   Thanks for the great video. had a question would appreciate your help. For computing the omega reliability of a from a measure, I was wondering whether I need to use the nfactors argument or not? I use function omega from psych package. I select the items for that subscale, and then use the omega function, however, output for total omega is different when I use the nfactor = 1 in the code. Should I write nfactor = 1 because it is one subscale of the measure or should I just left it empty? Thanks a lot!

   17/06/2022 at 12:58 PM

   ตอบกลับ
5. 

   Gabrijela Mikulić says:

   Can't believe I'm doing calculations for my Master's Thesis the same day this is uploaded. You are awesome!

   17/06/2022 at 12:58 PM

   ตอบกลับ
6. 

   statsguidetree says:

   Here is the rcode with notes:

   # Reliability is distinct from Validity but you cannot have a
   # valid instrument if it is not reliable.

   # Different types of reliability:
   # Test Re-test;
   # Parallel Form;
   # Inter-rater;
   # and Internal Consistency (e.g.,
   # Split-half Reliability, Chronbach's Alpa, etc.).

   ##### load dataset for example
   #install.packages("KernSmoothIRT")
   library(KernSmoothIRT)

   data(BDI)

   # remove NAs
   bdi<- data.frame(na.omit(BDIresponses))

   # Example of estimating Test Re-Test and Parallel Form
   # reliability with correlation of total scores between two tests
   # either the same test or parallel forms of the test.
   cor(rowSums(bdi),rowSums(bdi2))

   # Estimating inter-rater reliability the simpliest way is to
   # count number of times both raters agree and divide by total
   # items. A more percisie estimation accounts for agreement due
   # to chance is Cohen's Kappa (Cohen, 1960)

   library(psych)

   # Example take column 1 and 3 and assume each column represents
   # scores from different raters
   test1<-table(bdi$X1,bdi$X3)

   # run Cohen's Kappa
   cohen.kappa(test1, n.obs =239 )

   # Kappa just considers the matches on the main diagonal.
   # Weighted kappa considers off diagonal elements as well.
   # Weighted Kappa (Cohens, 1968) suggested to be reported for ordinal
   # scores (Bakeman & Gottman, 1997).

   # Many types of internal consistency reliability metrics
   # Chronbach's Alpha (Cronbach, 1951) is the most popular.

   # Benefits over test-retest, parallel-forms reliability,
   # and other intenal consistency metrics (e.g., split-half).
   # Alpha is the mean of all possible spit-half
   # reliabilities.

   # run alpha
   library(psych)
   alpha(bdi)

   # A Chronbach's Alpha of at least .7 would suggest good
   # reliability (Kline, 1999)

   # Alpha (Cronbach, 1951) = Guttman's lambda3 (Guttman, 1945)
   # Guttman's Lambda 6 (G6) is another type of internal consistency
   # metric and employes squared multiple correlations
   # it can differ from Alpha based on the presence of
   # multidimensionality.

   # Standardized alpha is based upon the correlations
   # rather than the covariances.

   # Alpha is a generalization of an earlier estimate
   # of reliability for tests with dichotomous items
   # developed by Kuder and Richardson, known as KR20,
   # and a shortcut approximation, KR21.

   # You can also account for reverse coded itmes with keys and setting
   # the item to negative. Not accounting for this could lead to
   # negative values.
   alpha(bdi, keys = c(1,-1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1))

   ##### Limitations of Alpha

   # Model that defines alpha is the essentially tau-equivalent model.
   # For internal consistency measurment there are three models as follows
   # from most restrictive to least:

   # Parallel model — the item true score variances, the item true score
   # means, and error variances are all constant.

   # Essentially tau-equivalent model — the item true score variances are
   # constant, but the item true score means and error variances can vary.

   # Congeneric model — the item true score variances, the item true score
   # means, and error variances can all vary.

   # The model assumptions are based on the Classical Test Theory (CTT)
   # framework, where your score on some scale (X) is based on a true
   # score (T) plus measurment error (E). True score and error are always
   # unknown but can be estimated. Reliability coefficients (r_xx) can be
   # used to estimate true and error variance, r_xx=True Var/Total Var.

   # The assumptions required for alpha, that all items in the scale
   # have equal sensitivity, is likely untenable in practice.
   # Additionally, most scales have some degree of multidimensionality,
   # which further violate unidimesnionality assumption required for
   # alpha.

   # Violation of these assumptions cause alpha to underestimate
   # internal consistency when errors for items are uncorrelated,
   # but they can overestimate internal consistency when errors
   # for items are very correlated and or scale length
   # is increased (Graham, 2006).

   # Alternatively, omega follows the conogeneric and has more
   # relaxed assumptions resulting in more accurate measure
   # of internal consitency.

   # Though some degree of multidimesionality is may be expected,
   # unidimesionality is assumption for both alpha and omega.
   # It is advisied that scales designed to measure multiple
   # factors should be divided into subscales and alpha or
   # omega should be calculated for each subscale (Dunn, Baguley,
   # & Brunsden, 2014).

   # When all assumptions of Alpha are met, alpha and omega are equivalent.
   # It is suggested that omega should be used when multidimensionality
   # is present (Chen et al., 2012, p. 228) and many researchers encourage
   # increase use of Omega over Alpha (e.g., Dunn et al., 2014;
   # Schweizer, 2011)

   # There should be a single latent variable common to most if not
   # all items in the scale in order to employ alpha/omega.

   # Alpha assumes equivalent loading on a single factor; however, this
   # assumption may be untenable in practical applications. Omega
   # allows factors loadings to vary.

   # Omega function omega() employs an exploratory factor analysis
   # and provides reliability estimates based on the general
   # and total factor saturation.

   # run omega
   library(psych)
   omega(bdi)

   # Omega is calculated by performing a factor analysis, the lower
   # level factors are rotated obliquely, then from the correlation
   # matrix produced one general factor is calculated and the
   # Shmid-Leiman transformation is used to find the item loadings
   # onto the general factor.

   # Types of omega coefficents:

   # Omega Total estimates the percision of a scale in measuring
   # multiple subscales as a multidimesnsional scale. Based on
   # sum of squared loadings on all factors.

   # Omega Hierarchical estimates the percision of a scale in measuring
   # one general/overall construct i.e. to what degree does a single
   # construct explain test score variance. Based on sum of square
   # loadings on the general factor (g) only.

   # Omega asymptotic is the Omega Hieraarchical calculated for an
   # infinite long test while maintaining the structure of the
   # scale

   # The difference is mainly that omega_t gives an reliability
   # estimate of the overall variance in the data that is due to
   # a general factor and lower level factors. The omega_h is
   # a reliability estimate for the variance that is due to the
   # general factor only.

   # Explained Common Variance (ECV) is the ratio of the general
   # factor eigen value to the sum of all of the eigen values.

   # Omega subset, each item is assigned to a group.
   # Omega total is the amount of within group variance accounted
   # for by the general and group factors omega general is the
   # amount of within group variance accounted for the general
   # factor only.

   # Note.Different types of reliability measure different facets
   # of reliability.

   17/06/2022 at 12:58 PM

   ตอบกลับ