Objective: The authors determine the reliability, validity, and usefulness of the Objective Structured Clinical Examination (OSCE) in the evaluation of surgical residents.
Summary background data: Interest is increasing in using the OSCE as a measurement of clinical competence and as a certification tool. However, concerns exist about the reliability, feasibility, and cost of the OSCE. Experience with the OSCE in postgraduate training programs is limited.
Methods: A comprehensive 38-station OSCE was administered to 56 surgical residents. Residents were grouped into three levels of training; interns, junior residents, and senior residents. The reliability of the examination was assessed by coefficient alpha; its validity, by the construct of experience. Differences between training levels and in performance on the various OSCE problems were determined by a three-way analysis of variance with two repeated measures and the Student-Newman-Keuls post hoc test. Pearson correlations were used to determine the relationship between OSCE and American Board of Surgery In-Training Examination (ABSITE) scores.
Results: The reliability of the OSCE was very high (0.91). Performance varied significantly according to level of training (postgraduate year; p < 0.0001). Senior residents performed best, and interns performed worst. The OSCE problems differed significantly in difficulty (p , 0.0001). Overall scores were poor. Important and specific performance deficits were identified at all levels of training. The ABSITE clinical scores, unlike the basic science scores, correlated modestly with the OSCE scores when level of training was held constant.
Conclusion: The OSCE is a highly reliable and valid clinical examination that provides unique information about the performance of individual residents and the quality of postgraduate training programs.