Development and Validation of a Predictive Model of the Risk of Pediatric Septic Shock Using Data Known at the Time of Hospital Arrival

Abstract

Objective: To derive and validate a model of risk of septic shock among children with suspected sepsis, using data known in the electronic health record at hospital arrival.

Study design: This observational cohort study at 6 pediatric emergency department and urgent care sites used a training dataset (5 sites, April 1, 2013, to December 31, 2016), a temporal test set (5 sites, January 1, 2017 to June 30, 2018), and a geographic test set (a sixth site, April 1, 2013, to December 31, 2018). Patients 60 days to 18 years of age in whom clinicians suspected sepsis were included; patients with septic shock on arrival were excluded. The outcome, septic shock, was systolic hypotension with vasoactive medication or ≥30 mL/kg of isotonic crystalloid within 24 hours of arrival. Elastic net regularization, a penalized regression technique, was used to develop a model in the training set.

Results: Of 2464 included visits, septic shock occurred in 282 (11.4%). The model had an area under the curve of 0.79 (0.76-0.83) in the training set, 0.75 (0.69-0.81) in the temporal test set, and 0.87 (0.73-1.00) in the geographic test set. With a threshold set to 90% sensitivity in the training set, the model yielded 82% (72%-90%) sensitivity and 48% (44%-52%) specificity in the temporal test set, and 90% (55%-100%) sensitivity and 32% (21%-46%) specificity in the geographic test set.

Conclusions: This model estimated the risk of septic shock in children at hospital arrival earlier than existing models. It leveraged the predictive value of routine electronic health record data through a modern predictive algorithm and has the potential to enhance clinical risk stratification in the critical moments before deterioration.

Keywords: diagnosis; emergency medicine; machine learning; prediction; sepsis.

Figure 1.

Mean deviance of cross-validated model compared with the number of variables included in the model.

Figure 2.

Study flow diagram.

Figure 3.

Receiver operating characteristic plots in the training and test sets.

Figure 4.

Calibration plots. A, Training set comparison of predicted proportion of patients with shock vs observed proportion of patients with shock, within deciles of risk (slope = 1.31; intercept = −0.04). B, Temporal test set comparison of predicted proportion of patients with shock vs observed proportion of patients with shock, within deciles of risk (slope = 1.28; intercept = 0.02). C, Geographic test set comparison of predicted proportion of patients with shock vs observed proportion of patients with shock, within tertiles of risk. Tertiles were used because the numbers were small in this dataset (slope = 1.35; intercept = −0.07).

Figure 5.

Precision recall curves. Precision indicates positive predictive value, and recall indicates sensitivity, and a model with perfect discrimination would follow the right upper borders of the plot. Performance of a model with no discrimination would equal the prevalence of the outcome in the dataset, indicated by the horizontal grey dashed line. A, Training set precision recall curve. Precision represents positive predictive value and recall represents sensitivity. B, Temporal test set precision recall curve. Precision represents positive predictive value and recall represents sensitivity. C, Geographic test set precision recall curve. Precision represents positive predictive value and recall represents sensitivity.