The development of an independent rater system to assess residents' competence in invasive procedures

Abstract

Purpose: To design an independent rater (IR) direct observation system to monitor invasive procedures performed by residents in the hospital setting.

Method: The authors recruited, trained, and tested nonphysicians to become IRs for an Agency for Healthcare Research and Quality-funded study evaluating the impact of partial task simulation training of ultrasound-guided central venous catheter (CVC) insertion on skills transfer at a major academic medical center. IR applicants completed four hours of training: a two-hour didactic session and a two-hour testing session, including observation of 5 of 10 choreographed CVC insertion videotapes and completion of a 50-data-point procedural checklist. Eligibility to be hired as an IR included timing the procedure accurately, detecting technical errors and complications, and completing the procedural checklist accurately.

Results: Thirty-eight IR trainees completed the training module and videotape examinations. Twenty-seven (71%) trainees met criteria to be hired IRs by accurately assessing the duration of the procedure to within one minute, validating the checklist to within 95% accuracy, and detecting technical errors/complications to within a 3% margin of error. The authors found no association between educational level and hired status, and all 13 IRs assessed after the study had maintained their skills.

Conclusions: Recent innovations in procedural training with partial task simulation trainers necessitate developing methods to measure skills transfer from the simulator to the clinical setting. This description of a nonphysician IR direct observation system for CVC insertion offers a feasible tool that may be generalized to monitoring other invasive procedures.