Does success on the American Board of Surgery general surgery qualifying examination guarantee certifying examination success?

Abstract

Objective: The purpose of this study was to explore the relationship between qualifying examination (QE) and certifying examination (CE) results and to determine whether an appropriate cut-point on the QE would predict success on the CE.

Design: The scaled American Board of Surgery (ABS) QE scores of all first-time examinees from 2006 to 2010 were retrospectively matched to their first-time CE pass/fail decisions. Contingency tables illustrating the QE-CE relationship were constructed and appropriate correlational statistics were computed. A receiver operating characteristic (ROC) curve analysis was constructed, with sensitivity and 1-specificity calculated for each possible QE cut-point used to predicted CE pass/fail classifications. Additionally, the area under the curve (AUC) was calculated.

Participants: All first-time American Board of Surgery examinees for the Surgery Qualifying Examination from 2006 to 2010.

Results: A total of 4385 surgeons were analyzed, with QE scores averaging 82.1 ± 5.8 (range, 58-99) and 82.8% of surgeons passing the CE on their first attempt. Contingency tables suggest a moderate relationship between QE and CE performance, although correlation indexes are low (phi = 0.13, point-biserial = 0.23). For the ROC analysis, the AUC = 0.674 (95% CI; 0.654-0.695) provides a better than chance pass/fail classification (p < 0.001), yet does not meet the minimum threshold for acceptability as a predictive test. No QE cut-point accurately predicted CE pass/fail decisions.

Conclusions: While a moderate relationship between QE scores and CE performance is evident, correlations suggest that the 2 examinations measure different abilities. The low AUC value on the ROC analysis, along with poor predictability at all possible cut-points, show that no appropriate cut-point on the QE predicts success on the CE. These data add to the validity of both tests by providing evidence that distinct latent traits are being measured by both tests.