Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 48 (2), 210-217

Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019

Affiliations

Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019

Matthew A Reyna et al. Crit Care Med.

Abstract

Objectives: Sepsis is a major public health concern with significant morbidity, mortality, and healthcare expenses. Early detection and antibiotic treatment of sepsis improve outcomes. However, although professional critical care societies have proposed new clinical criteria that aid sepsis recognition, the fundamental need for early detection and treatment remains unmet. In response, researchers have proposed algorithms for early sepsis detection, but directly comparing such methods has not been possible because of different patient cohorts, clinical variables and sepsis criteria, prediction tasks, evaluation metrics, and other differences. To address these issues, the PhysioNet/Computing in Cardiology Challenge 2019 facilitated the development of automated, open-source algorithms for the early detection of sepsis from clinical data.

Design: Participants submitted containerized algorithms to a cloud-based testing environment, where we graded entries for their binary classification performance using a novel clinical utility-based evaluation metric. We designed this scoring function specifically for the Challenge to reward algorithms for early predictions and penalize them for late or missed predictions and for false alarms.

Setting: ICUs in three separate hospital systems. We shared data from two systems publicly and sequestered data from all three systems for scoring.

Patients: We sourced over 60,000 ICU patients with up to 40 clinical variables for each hour of a patient's ICU stay. We applied Sepsis-3 clinical criteria for sepsis onset.

Interventions: None.

Measurements and main results: A total of 104 groups from academia and industry participated, contributing 853 submissions. Furthermore, 90 abstracts based on Challenge entries were accepted for presentation at Computing in Cardiology.

Conclusions: Diverse computational approaches predict the onset of sepsis several hours before clinical recognition, but generalizability to different hospital systems remains a challenge.

Conflict of interest statement

Drs. Reyna, Jeter, Nemati, and Clifford are partially funded by the National Science Foundation under award number 1822378 (Leveraging Heterogeneous Data Across International Borders in a Privacy Preserving Manner for Clinical Deep Learning). Mr. Shashikumar and Dr. Nemati are also funded by the National Institutes of Health (NIH) award number K01ES025445. Drs. Reyna, Josef, Jeter, Westover, Clifford, and Sharma received support for article research from the NIH. Dr. Josef’s institution received funding from NIH (T-32 Grant Trainee: T32GM095442-09) and Henry M. Jackson Foundation for role as a post-doctoral researcher for the Surgical Critical Care Institute, www.sc2i.org, funded through the Department of Defense’s Defense Health Program—Joint Program Committee 6/Combat Casualty Care (Uniformed Services University of the Health Sciences HT9404-13-1-0032 and HU0001-15-2-0001) and was supported by a grant from the NIH, United States (NIH grant: 5T32GM095442-09). Dr. Clifford’s institution received funding from the Gordon and Betty Moore Foundation and NIH and he received cloud credits from Google Cloud. Dr. Sharma’s institution received funding from the Gordon and Betty Moore Foundation, and he received funding from Google (travel reimbursement for a talk at a seminar). Dr. Sharma and the development of the cloud-based scoring system were partially supported by the National Cancer Institute (U24CA215109). The remaining authors have disclosed that they do not have any potential conflicts of interest.

Figures

Figure 1.
Figure 1.
Densities of vital sign (rows 1-8) and laboratory value (rows 9-34) entries (fraction of non-empty entries) in the shared and hidden datasets for hospital systems A, B, and C.
Figure 2.
Figure 2.
Diagrams of utility of positive and negative predictions for sepsis and non-septic patients; the time tsepsis = 48 of sepsis onset is given as an example.
Figure 3.
Figure 3.
Comparison of each algorithm’s AUROC and utility scores on test data from hospital systems A, B, and C, where we shared training data for hospital systems A and B but not for hospital system C. A, Comparison of each algorithm’s area under the receiver operating characteristic curve (AUROC) and utility scores on test sets A, B, and C. B, Comparison of each algorithm’s AUROC and utility scores on test sets A and B. C, Comparison of each algorithm’s AUROC and utility scores on test sets A and C. D, Comparison of each algorithm’s AUROC and utility scores on test sets B and C. E, Ranked performance of the final algorithms on test sets A, B, and C. Red indicates a high overall ranking across all three databases, and blue indicates a low overall ranking. Lines from top to bottom indicate how the individual algorithm ranking changed when considering the performance on each database. Algorithms that performed well on test sets A and B generally performed relatively poorly on test set C.

Similar articles

See all similar articles

References

    1. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis: For the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315:762–774 - PMC - PubMed
    1. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315:801–810 - PMC - PubMed
    1. Shankar-Hari M, Phillips GS, Levy ML, et al. Developing a new definition and assessing new clinical criteria for septic shock: For the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315:775–787 - PMC - PubMed
    1. Centers for Disease Control and Prevention: Sepsis. Cited 23 August 2016. https://www.cdc.gov/sepsis/datareports/index.html,. Accessed February 1 2019.
    1. World Health Organization: Sepsis. Cited 19 April 2018s. Available at: https://www.who.int/news-room/fact-sheets/detail/sepsis. Accessed February 1 2019.
Feedback