Norming a Written Communication Rubric in a Graduate Health Science Course

J Allied Health. 2016 Fall;45(3):e37-42.

Abstract

Background: There is growing interest in the use of rubrics to assess written work. This study aimed to determine whether or not the norming of a written communication rubric improved scoring consistency among clinical faculty in a critical thinking course. The benefits of a formalized norming process are described.

Methods: Faculty-raters were trained to apply the rubric to a signature assignment while participating in calibration workshops. For each rubric criterion, faculty examined whether or not heightened congruence in scoring resulted from the training. Inter-rater reliability was determined after raters independently scored de-identified essays.

Results: Pre-workshop intra-class correlations (ICCs) were acceptable (i.e., >0.7) for three of five rubric criteria. Post-workshop ICCs for only two criteria were acceptable: disciplinary conventions, and sources and evidence. Rater attrition and lag-time between calibration and post-workshop activities likely contributed to reduced consistency.

Discussion: The rubric was useful for discriminating writing proficiency. Norming led to revision of the signature assignment, the rubric design, and a need for writing workshops. These changes will result in better student preparation for composing evidence-informed essays. Less-rigid approaches are worthy of future exploration.

MeSH terms

  • Communication*
  • Education, Graduate*
  • Educational Measurement / methods
  • Guidelines as Topic
  • Humans
  • Information Literacy
  • Reproducibility of Results
  • Writing*