Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm

Abstract

Background: Surgical complications have tremendous consequences and costs. Complication detection is important for quality improvement, but traditional manual chart review is burdensome. Automated mechanisms are needed to make this more efficient. To understand the generalizability of a machine learning algorithm between sites, automated surgical site infection (SSI) detection algorithms developed at one center were tested at another distinct center.

Study design: NSQIP patients had electronic health record (EHR) data extracted at one center (University of Minnesota Medical Center, Site A) over a 4-year period for model development and internal validation, and at a second center (University of California San Francisco, Site B) over a subsequent 2-year period for external validation. Models for automated NSQIP SSI detection of superficial, organ space, and total SSI within 30 days postoperatively were validated using area under the curve (AUC) scores and corresponding 95% confidence intervals.

Results: For the 8,883 patients (Site A) and 1,473 patients (Site B), AUC scores were not statistically different for any outcome including superficial (external 0.804, internal [0.784, 0.874] AUC); organ/space (external 0.905, internal [0.867, 0.941] AUC); and total (external 0.855, internal [0.854, 0.908] AUC) SSI. False negative rates decreased with increasing case review volume and would be amenable to a strategy in which cases with low predicted probabilities of SSI could be excluded from chart review.

Conclusions: Our findings demonstrated that SSI detection machine learning algorithms developed at 1 site were generalizable to another institution. SSI detection models are practically applicable to accelerate and focus chart review.

Figure 1.

Overview of experimental approach for internal and external validation of algorithms for surgical site infection detection. EHR, electronic health record; UCSF, University of California, San Francisco; UMMC, University of Minnesota Medical Center.

Figure 2.

False negative rate (FNR) of 3 models vs percentage of case reviews based on predicted probabilities of surgical site infection (SSI) development. (A) Superficial SSI; (B) organ space SSI; (C) total SSI. FNR is calculated by applying the models internally on University of Minnesota Medical Center (UMMC, Site A) dataset and externally on University of California, San Francisco (UCSF, Site B) dataset. Generally, the results suggest that the developed models have lower FNR on the Site A dataset.