The Impact of Surgeon Experience on Script Concordance Test Scoring

J Surg Res. 2021 May 5;265:265-271. doi: 10.1016/j.jss.2021.03.057. Online ahead of print.

Abstract

Objective: The Script Concordance Test (SCT) is a test of clinical decision-making that relies on an expert panel to create its scoring key. Existing literature demonstrates the value of specialty-specific experts, but the effect of experience among the expert panel is unknown. The purpose of this study was to explore the role of surgeon experience in SCT scoring.

Design: An SCT was administered to 29 general surgery residents and 14 staff surgeons. Staff surgeons were stratified as either junior or senior experts based on years since completing residency training (<15 versus >25 years). The SCT was scored using the full expert panel, the senior panel, the junior panel, and a subgroup junior panel in practice <5 years. A one-way ANOVA was used to compare the scores of first (R1) and fifth (R5) year residents using each scoring scheme. Cognitive interviews were analyzed for differences between junior and senior expert panelist responses.

Results: There was no statistically significant difference between the mean score of six R1s and five R5s using the full expert panel (R1 69.08 versus R5 67.06, F1,9 = 0.10, P = 0.76), the junior panel (R1 66.73 versus R5 62.50, F1,9 = 0.35, P = 0.57), or the subgroup panel in practice <5 years (R1 61.07 versus R5 58.79, F1,9 = 0.18, P = 0.75). However, the average score of R1s was significantly lower than R5s when using the senior faculty panel (R1 52.04 versus R5 63.26, F1,9 = 26.90, P = 0.001). Cognitive interview data suggests that some responses of junior experts demonstrate less confidence than those of senior experts.

Conclusions: SCT scores are significantly affected by the responses of the expert panel. Expert differences between first and fifth year residents were only demonstrated when using an expert panel consisting of senior faculty members. Confidence may play a role in the response selections of junior experts. When constructing an SCT expert panel, consideration must be given to the experience of panel members.

Keywords: Clinical decision-making; Experience; Expert panel; Script concordance test.