Use of an error-focused checklist to identify incompetence in lumbar puncture performances

Med Educ. 2015 Oct;49(10):1004-15. doi: 10.1111/medu.12809.


Context: Checklists are commonly used in the assessment of procedural competence. However, on most checklists, high scores are often unable to rule out incompetence as the commission of a few serious procedural errors typically results in only a minimal reduction in performance score. We hypothesised that checklists constructed based on procedural errors may be better at identifying incompetence.

Objectives: This study sought to compare the efficacy of an error-focused checklist and a conventionally constructed checklist in identifying procedural incompetence.

Methods: We constructed a 15-item error-focused checklist for lumbar puncture (LP) based on input from 13 experts in four Canadian academic centres, using a modified Delphi approach, over three rounds of survey. Ratings of 18 video-recorded performances of LP on simulators using the error-focused tool were compared with ratings obtained using a published conventional 21-item checklist. Competence/incompetence decisions were based on global assessment. Diagnostic accuracy was estimated using the area under the curve (AUC) in receiver operating characteristic analyses.

Results: The accuracy of the conventional checklist in identifying incompetence was low (AUC 0.11, 95% confidence interval [CI] 0.00-0.28) in comparison with that of the error-focused checklist (AUC 0.85, 95% CI 0.67-1.00). The internal consistency of the error-focused checklist was lower than that of the conventional checklist (α = 0.35 and α = 0.79, respectively). The inter-rater reliability of both tools was high (conventional checklist: intraclass correlation coefficient [ICC] 0.99, 95% CI 0.98-1.00; error-focused checklist: ICC 0.92, 95% CI 0.68-0.98).

Conclusions: Despite higher internal consistency and inter-rater reliability, the conventional checklist was less accurate at identifying procedural incompetence. For assessments in which it is important to identify procedural incompetence, we recommend the use of an error-focused checklist.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Checklist*
  • Clinical Competence*
  • Humans
  • Medical Errors*
  • Observer Variation
  • Spinal Puncture* / standards
  • Video Recording