Can machine learning complement traditional medical device surveillance? A case study of dual-chamber implantable cardioverter-defibrillators

Abstract

Background: Machine learning methods may complement traditional analytic methods for medical device surveillance.

Methods and results: Using data from the National Cardiovascular Data Registry for implantable cardioverter-defibrillators (ICDs) linked to Medicare administrative claims for longitudinal follow-up, we applied three statistical approaches to safety-signal detection for commonly used dual-chamber ICDs that used two propensity score (PS) models: one specified by subject-matter experts (PS-SME), and the other one by machine learning-based selection (PS-ML). The first approach used PS-SME and cumulative incidence (time-to-event), the second approach used PS-SME and cumulative risk (Data Extraction and Longitudinal Trend Analysis [DELTA]), and the third approach used PS-ML and cumulative risk (embedded feature selection). Safety-signal surveillance was conducted for eleven dual-chamber ICD models implanted at least 2,000 times over 3 years. Between 2006 and 2010, there were 71,948 Medicare fee-for-service beneficiaries who received dual-chamber ICDs. Cumulative device-specific unadjusted 3-year event rates varied for three surveyed safety signals: death from any cause, 12.8%-20.9%; nonfatal ICD-related adverse events, 19.3%-26.3%; and death from any cause or nonfatal ICD-related adverse event, 27.1%-37.6%. Agreement among safety signals detected/not detected between the time-to-event and DELTA approaches was 90.9% (360 of 396, k=0.068), between the time-to-event and embedded feature-selection approaches was 91.7% (363 of 396, k=-0.028), and between the DELTA and embedded feature selection approaches was 88.1% (349 of 396, k=-0.042).

Conclusion: Three statistical approaches, including one machine learning method, identified important safety signals, but without exact agreement. Ensemble methods may be needed to detect all safety signals for further evaluation during medical device surveillance.

Keywords: implanted cardioverter–defibrillator; methodology; surveillance.

Figure 1

PS-ML algorithm to determine the regularization parameter. Abbreviations: PS, propensity score; ML, machine learning.

Figure 2

Cohort-derivation flowchart. Abbreviations: ICD, implantable cardioverter-defibrillator; MPN, Medicare provider number.

Figure 3

Kaplan–Meier survival curve. Note: Freedom from death or nonfatal ICD-related adverse events among Medicare fee-for-service beneficiaries undergoing dual-chamber ICD implantation, 2006–2010. Abbreviation: ICD, implantable cardioverter–defibrillator.

Figure 4

Cumulative implantation volume for the eleven dual-chamber ICDs. Note: ICDs implanted 2,000 times or more among Medicare fee-for-service beneficiaries, 2006–2010. Abbreviation: ICDs, implantable cardioverter–defibrillators.

Figure 5

Time-to-event, DELTA, and embedded feature selection approaches for safety-signal detection among eleven commonly used dual-chamber ICDs, 2006–2010. Note: a, Rate higher than control; b, rate lower than control. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICDs, implantable cardioverter–defibrillators.

Figure 6

Time-to-event, DELTA, and embedded feature selection methods for safety-signal detection for death among eleven commonly used dual-chamber ICDs, 2006–2010. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICDs, implantable cardioverter–defibrillators.

Figure 7

Time-to-event, DELTA, and embedded feature selection methods for safety-signal detection for any nonfatal adverse event among eleven commonly used dual-chamber ICDs, 2006–2010. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICDs, implantable cardioverter–defibrillators.

Figure 8

Time-to-event, DELTA, and embedded feature selection methods for safety-signal detection for death or any adverse event among eleven commonly used dual-chamber ICDs, 2006–2010. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICDs, implantable cardioverter–defibrillators.

Figure 9

Medical device performance for ICD 1 relative to propensity-matched control devices. Notes: Measured by composite end point of death or nonfatal ICD-related adverse events, assessed using (A) DELTA and (B) embedded feature selection. “Bad” indicates quarters where device performance was significantly worse (more adverse events) relative to propensity-matched control devices; “Good” indicates quarters where device performance was significantly better (less adverse events) relative to propensity-matched control devices; “Imbalanced” indicates quarters where at least one covariate was imbalanced among the propensity-matched groups. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICD, implantable cardioverter–defibrillator.

Figure 10

Medical device performance for ICD 5 relative to propensity-matched control devices. Notes: Measured by composite end point of death or nonfatal ICD-related adverse events, assessed using (A) DELTA and (B) embedded feature selection. “Bad” indicates quarters where device performance was significantly worse (more adverse events) relative to propensity-matched control devices; “Good” indicates quarters where device performance was significantly better (less adverse events) relative to propensity-matched control devices; “Imbalanced” indicates quarters where at least one covariate was imbalanced among the propensity-matched groups. Abbreviations: DELTA, Data Extraction and Longitudinal Trend Analysis; ICD, implantable cardioverter–defibrillator.