Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity

J Clin Epidemiol. 2000 May;53(5):499-503. doi: 10.1016/s0895-4356(99)00174-2.


Paradoxical effects of bias and prevalence on the kappa coefficient are examined using the concepts of sensitivity and specificity. Results that appear paradoxical when viewed as a 2 x 2 table of frequencies do not appear paradoxical when viewed as a pair of sensitivity and specificity measures where each observer is treated as a predictor of the other observer. An adjusted kappa value can be obtained from these sensitivity/specificity measures but simulation studies indicate that it would result in substantial overestimation of reliability when bias or prevalence effects are observed. It is suggested that investigators concentrate on obtaining populations with trait prevalence near 50% rather than searching for statistical indices to rescue or excuse inefficient experiments.

MeSH terms

  • Bias*
  • Computer Simulation
  • Data Interpretation, Statistical
  • Humans
  • Observer Variation*
  • Odds Ratio
  • Prevalence*
  • ROC Curve
  • Sensitivity and Specificity*