Examining the Effects of Narrative Commentary on Evaluators' Summative Assessments of Resident Performance

Eval Health Prof. 2020 Sep;43(3):159-161. doi: 10.1177/0163278718820415. Epub 2018 Dec 26.

Abstract

Anchor-based, end-of-shift ratings are commonly used to conduct performance assessments of resident physicians. These performance evaluations often include narrative assessments, such as solicited or "free-text" commentary. Although narrative commentary can help to create a more detailed and specific assessment of performance, there are limited data describing the effects of narrative commentary on the global assessment process. This single-group, observational study examined the effect of narrative comments on global performance assessments. A subgroup of the clinical competency committee, blinded to resident identity, assigned a single, consensus-based performance score (1-6) to each resident based solely on end-of-shift milestone scores. De-identified narrative comments from end-of-shift evaluations were then included and the process was repeated. We compared milestone-only scores to milestone plus narrative commentary scores using a nonparametric sign test. During the study period, 953 end-of-shift evaluations were submitted on 41 residents. Of these, 535 evaluations included free-text narrative comments. In 17 of the 41 observations, performance scores changed after the addition of narrative comments. In two cases, scores decreased with the addition of free-text commentary. In 15 cases, scores increased. The frequency of net positive change was significant (p = .0023). The addition of narrative commentary to anchor-based ratings significantly influenced the global performance assessment of Emergency Medicine residents by a committee of educators. Descriptive commentary collected at the end of shift may inform more meaningful appraisal of a resident's progress in a milestone-based paradigm. The authors recommend clinical training programs collect unstructured narrative impressions of residents' performance from supervising faculty.

Keywords: comments; evaluation; milestones; narrative; performance assessment.

Publication types

  • Observational Study

MeSH terms

  • Clinical Competence / standards*
  • Educational Measurement / methods*
  • Emergency Medicine / education*
  • Humans
  • Internship and Residency / standards*
  • Narration*
  • Prospective Studies