Simulation-Based Assessment of Critical Care "Front-Line" Providers

Crit Care Med. 2018 Jun;46(6):e516-e522. doi: 10.1097/CCM.0000000000003073.


Objectives: Develop a standardized simulation method to assess clinical skills of ICU providers.

Design: Simulation assessment.

Setting: Simulation laboratory.

Subjects: Residents, Critical Care Medicine fellows, acute care nurse practitioner students.

Interventions: Performance scoring in scenarios from multiple Critical Care Medicine competency domains.

Measurements and main results: Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p < 0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (< 5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively.

Conclusions: The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.

Publication types

  • Research Support, U.S. Gov't, P.H.S.

MeSH terms

  • Adult
  • Checklist
  • Clinical Competence* / standards
  • Clinical Decision-Making
  • Critical Care* / standards
  • Female
  • Humans
  • Internship and Residency / standards
  • Male
  • Middle Aged
  • Nurse Practitioners / standards
  • Patient Simulation
  • Reproducibility of Results