Effect of Standardised Scoring Conventions on Inter-rater Reliability in the Endoscopic Evaluation of Crohn's Disease

J Crohns Colitis. 2016 Sep;10(9):1006-14. doi: 10.1093/ecco-jcc/jjw120. Epub 2016 Jul 6.


Background and aims: The Crohn's Disease Endoscopic Index of Severity [CDEIS] and Simplified Endoscopic Score for Crohn's Disease [SES-CD] demonstrate consistent overall intra- and inter-rater reliability. However, the reliability of some index items is relatively poor. We evaluated scoring conventions to improve the reliability of these items.

Methods: Five gastroenterologists with no previous experience scoring the CDEIS or SES-CD were trained on their use. A total of 65 video recordings of colonoscopies were scored blindly by each gastroenterologist before and after additional training on index scoring conventions. Intra-class correlation coefficients [ICCs] assessed the effect of application of these conventions on the reliability of the CDEIS, SES-CD, and a Global Evaluation of Lesion Severity [GELS] score.

Results: Following training on scoring conventions, inter-rater ICCs (95% confidence interval [CI]) for the total SES-CD score increased from 0.78 [0.71, 0.85] to 0.85 [0.79, 0.89]. The ICCs for the total CDEIS and GELS scores were not affected: corresponding inter-rater ICCs were 0.74 [0.65, 0.81] and 0.49, [0.38, 0.61] before and 0.73 [0.65, 0.81] and 0.53 [0.42, 0.64] following application of scoring conventions. Estimations of ulcer depth, surface area, anatomical location, and stenosis were important sources of variability.

Conclusions: Use of scoring conventions previously developed by expert central readers enhanced the reliability of the SES-CD but did not similarly affect the CDEIS or GELS. As the SES-CD is more likely to be reliable than the CDEIS and can be optimised with targeted training, it is the preferred instrument for use in clinical trials.

Keywords: Crohn’s disease; endoscopic indices; training.

Publication types

  • Evaluation Study

MeSH terms

  • Adolescent
  • Adult
  • Aged
  • Aged, 80 and over
  • Clinical Decision-Making / methods*
  • Colonoscopy*
  • Crohn Disease / diagnostic imaging*
  • Crohn Disease / pathology
  • Education, Medical, Continuing
  • Female
  • Gastroenterology / education
  • Humans
  • Male
  • Middle Aged
  • Observer Variation
  • Ontario
  • Reproducibility of Results
  • Severity of Illness Index*
  • Single-Blind Method
  • Video Recording
  • Young Adult