Using Precept-Assist® to predict performance on the American Board of Family Medicine In-Training Examination

Abstract

Background and objectives: Precept-Assist® (PA) is a computer-based program developed by the Virtua Family Medicine Residency where residents receive a score on a Likert-type scale from an attending for each precept based on their knowledge base. The purpose of this study was to attempt to validate this program for precepting family medicine residents.

Methods: This was a validation study. PA and American Board of Family Medicine (ABFM) In-Training Exam (ITE) scores for all residents from a community-based family medicine residency between the years 2002 and 2011 were included (n=216). Pearson correlation coefficients were calculated between PA scores for the second quarter of the academic year (October 1 to December 31) and scores on the ITE. An ROC curve was also created to determine sensitivity and specificity for various PA scores in predicting residents scoring 500 or above on the ITE.

Results: The PA mean (SD) score was 5.18 (0.84) and the ITE mean (SD) score was 425.1 (87.6). The Pearson correlation coefficient between PA and ITE scores was 0.55, which is a moderately positive correlation. The AUC of the ROC curve was 0.783 (95% CI 0.704-0.859). A PA score of 5.5 (between the level of a PGY-2 and PGY-3) was 72% sensitive and 77% specific for scoring 500 or above on the ITE with a positive LR of 3.12.

Conclusions: There is a significant correlation between PA scores and ABFM In-Training Exam scores. PA is a valid screening tool that can be used as a predictor for future performance in Family Medicine In-Training exams.