Reliability and Validity of a Scoring Instrument for Clinical Performance During Pediatric Advanced Life Support Simulation Scenarios

Resuscitation. 2010 Mar;81(3):331-6. doi: 10.1016/j.resuscitation.2009.11.011. Epub 2010 Jan 4.

Abstract

Aim: To assess the reliability and validity of scoring instruments designed to measure clinical performance during simulated resuscitations requiring the use of Pediatric Advanced Life Support (PALS) algorithms.

Methods: Pediatric residents were invited to participate in an educational trial involving simulated resuscitations that employ PALS algorithms. Each subject participated in a session comprised of four scenarios (asystole, dysrhythmia, respiratory arrest, shock). Video-recorded sessions were independently reviewed and scored by four raters using instruments designed to measure performance in terms of timing, sequence, and quality. Validity was assessed by two-factor analysis of variance with postgraduate year (PGY-1 versus PGY-2) as an independent variable. Reliability was assessed by calculation of overall interrater reliability (IRR) as well as a generalizability study to estimate variance components of individual measurement facets (scenarios, raters) and associated interactions.

Results: 20 subjects were scored by four raters. Based on a two-factor ANOVA, PGY-2s outperformed PGY-1s (p<0.05); significant differences in difficulty existed between the four scenarios, with dysrhythmia scores being the lowest. Overall IRR was high (0.81) and most variance could be attributed to subject (17%), scenario (13%), and the interaction between subject and scenario (52%); variance attributable to rater was minimal (1.4%).

Conclusions: The instruments assessed in this study measure clinical performance during PALS scenarios in a reliable and valid manner. Measurement error could be minimized further through the use of additional scenarios but additional raters, for a given scenario, would not improve reliability. Further studies should assess validity of measurement with respect to actual clinical performance during resuscitations.

Publication types

  • Research Support, Non-U.S. Gov't
  • Validation Study

MeSH terms

  • Algorithms
  • Analysis of Variance
  • Arrhythmias, Cardiac / therapy
  • Child
  • Clinical Competence / standards*
  • Factor Analysis, Statistical
  • Heart Arrest / therapy
  • Humans
  • Internship and Residency
  • Life Support Systems*
  • Patient Simulation*
  • Pediatrics / education*
  • Pediatrics / methods
  • Random Allocation
  • Reproducibility of Results
  • Respiratory Insufficiency / therapy
  • Resuscitation / standards*
  • Shock / therapy
  • Videotape Recording