Inter rater agreement computation

48 views Asked by At

In my study we let 10 human raters rate 2 books on 3 aspects (accuracy, clarity, simplicity). I am trying to calculate the inter-rater agreement but facing issue which I am unable to resolve. I am new to R programming. here is a reproducible example and i request your help!

data <- data.frame(
  rater = c("rater-1", "rater-2", "rater-3"),
  clarity = c(3, 4,4),
  simplicity = c(5, 5,4),
  accuracy = c(2, 4,5)
)

# Convert the data frame to a matrix
ratings <- as.matrix(data[, -1])

# Calculate Fleiss' Kappa for each dimension
clarity_kappa <- kappa2(ratings[, "clarity"])
simplicity_kappa <- kappa2(ratings[, "simplicity"])
accuracy_kappa <- kappa2(ratings[, "accuracy"])

# Output the results
print(paste("Clarity Kappa:", clarity_kappa$value))
print(paste("Simplicity Kappa:", simplicity_kappa$value))
print(paste("Accuracy Kappa:", accuracy_kappa$value))

Error:

clarity_kappa <- kappa2(ratings[, "clarity"]) Error in ratings[, 2]: subscript out of bounds

0

There are 0 answers