Assessment of an In Silico Mechanistic Model for Proarrhythmia Risk Prediction Under the CiPA Initiative

Abstract

The International Council on Harmonization (ICH) S7B and E14 regulatory guidelines are sensitive but not specific for predicting which drugs are pro-arrhythmic. In response, the Comprehensive In Vitro Proarrhythmia Assay (CiPA) was proposed that integrates multi-ion channel pharmacology data in vitro into a human cardiomyocyte model in silico for proarrhythmia risk assessment. Previously, we reported the model optimization and proarrhythmia metric selection based on CiPA training drugs. In this study, we report the application of the prespecified model and metric to independent CiPA validation drugs. Over two validation datasets, the CiPA model performance meets all pre-specified measures for ranking and classifying validation drugs, and outperforms alternatives, despite some in vitro data differences between the two datasets due to different experimental conditions and quality control procedures. This suggests that the current CiPA model/metric may be fit for regulatory use, and standardization of experimental protocols and quality control criteria could increase the model prediction accuracy even further.

Figure 1

The Comprehensive In Vitro Proarrhythmia Assay (CiPA) in silico model qualification procedure. Shown is the flowchart of the CiPA in silico model qualification process designed by the CiPA Steering Committee. The model training process includes model optimization and metric development using published human cardiomyocyte experimental data originally used for O'Hara Rudy (ORd) model development, and newly acquired in vitro drug block data against various cardiac currents for the 12 training compounds. This training process was performed prior to, and strictly separated from, the model validation process—where the model and metric predefined by the training data were used to predict the TdP risk of the 16 validation drugs using their in vitro data. The model performance measures (Validation Strategy/ Supplementary Text S1) to evaluate the prediction accuracy were also prespecified before the validation began.

Figure 2

The distribution of torsade metric scores for the 12 Comprehensive In Vitro Proarrhythmia Assay training drugs. For each of the training drugs, 2,000 torsade metric scores were calculated using the uncertainty quantification method developed previously.17 The 95% confidence interval and median point of the 2,000 torsade metric scores for each drug are shown as horizontal error bars in this figure. Threshold 1 and threshold 2 are calculated by ordinal logistic regression (see Supplementary Text S2) to separate the three TdP risk categories (red: high risk; blue: intermediate risk; green: low/no risk). The values of these thresholds are given in the main text. Drugs are sorted by the median values of their torsade metric scores in each dataset. Results shown are for the manual (a) and hybrid (b) training datasets, respectively.

Figure 3

The receiver operating characteristic (ROC)1 analysis to estimate the probability of ranking high‐risk or intermediate‐risk drugs above low‐risk drugs. For ROC1 analysis, high‐risk and intermediate‐risk drugs are combined into one category (high‐or‐intermediate), and 10,000 ROC curves are constructed by sampling the torsade metric score distributions for the manual (a) and hybrid (b) validation datasets, respectively. Left panel of a or b: one representative example of the 10,000 ROC curves and the corresponding area under the curve (AUC). Insert: the underlying ranking of the 16 drugs (X axis: rank of 1–16; drug names not shown on X axis for figure clarity) for this particular ROC curve according to their torsade metric scores (Y axis); L: low/no risk drugs; H‐or‐I: high‐or‐intermediate risk drugs. Note that H‐or‐I drugs (black) generally have a torsade metric score lower than L drugs (white), indicating a higher ranking of Torsade de Pointes risk. Right panel of a or b: Distribution of the AUCs across the 10,000 ROC curves.

Figure 4

The distribution of torsade metric score values for the 16 Comprehensive In Vitro Proarrhythmia Assay (CiPA) validation drugs. For each of the validation drugs, 2,000 torsade metric scores are calculated using the Uncertainty Quantification (UQ) method developed earlier17 to describe the probability distribution of the risk metric. The 95% confidence interval and median point of the 2,000 torsade metric scores for each drug are shown as horizontal error bars. Drugs are sorted according to their median torsade metric scores within each category in each dataset. Threshold 1 and threshold 2 are predefined by training, as in Figure 2. (a) Results for the manual validation dataset. (b) The hybrid validation dataset.

Figure 5

Comparison of the in vitro data for the outliers between the manual and hybrid datasets. The in vitro concentration‐dependent block data of the three outliers (incorrectly predicted by both ranking and classification) are shown for metoprolol on late sodium current (INaL) (a), disopyramide on INaL (b), and domperidone on L‐type calcium current (ICaL), and (c) from the manual (left) and hybrid (right) validation datasets, respectively. Circles: experimental data points for each cell. Solid line: fitting using median values of concentration of half inhibition and Hill coefficient. Gray band: 95% confidence interval of the fitting. Vertical dotted lines: the start and end of the concentration range (1–4× maximum free plasma concentration (C_max)) used for calculating torsade metric score. The mean block% as estimated by the fitted Hill equation curves between 1 and 4× C_max is labeled. Note that for INaL block (a and b), the manual dataset shows much less potency than the hybrid dataset, whereas for ICaL (c), the manual dataset shows much higher potency than the hybrid dataset. This explains the emergence of outliers unique to each dataset.