Benchmark of computational methods for predicting microRNA-disease associations

Abstract

Background: A series of miRNA-disease association prediction methods have been proposed to prioritize potential disease-associated miRNAs. Independent benchmarking of these methods is warranted to assess their effectiveness and robustness.

Results: Based on more than 8000 novel miRNA-disease associations from the latest HMDD v3.1 database, we perform systematic comparison among 36 readily available prediction methods. Their overall performances are evaluated with rigorous precision-recall curve analysis, where 13 methods show acceptable accuracy (AUPRC > 0.200) while the top two methods achieve a promising AUPRC over 0.300, and most of these methods are also highly ranked when considering only the causal miRNA-disease associations as the positive samples. The potential of performance improvement is demonstrated by combining different predictors or adopting a more updated miRNA similarity matrix, which would result in up to 16% and 46% of AUPRC augmentations compared to the best single predictor and the predictors using the previous similarity matrix, respectively. Our analysis suggests a common issue of the available methods, which is that the prediction results are severely biased toward well-annotated diseases with many associated miRNAs known and cannot further stratify the positive samples by discriminating the causal miRNA-disease associations from the general miRNA-disease associations.

Conclusion: Our benchmarking results not only provide a reference for biomedical researchers to choose appropriate miRNA-disease association predictors for their purpose, but also suggest the future directions for the development of more robust miRNA-disease association predictors.

Keywords: Benchmarking test; Disease miRNA prediction; miRNA-disease association.

Fig. 1

Overall performance of 36 miRNA-disease association predictors on the benchmarking datasets. a The flow chart depicting the inclusion/exclusion criterion for the predictors. The count of predictors included/excluded at each step is indicated by the number in the parentheses, and the fractions of predictors trained with different training datasets are depicted by the associated pie charts. b Precision-recall curves of the top ten predictors in terms of AUPRC on the ALL benchmarking dataset. c The statistics of correctly predicted miRNA-disease association pairs among the top 100, top 500, top 1000, and top 5% highly scored predictions on the ALL benchmarking dataset. d Precision-recall curves of the top ten predictors in terms of AUPRC on the CAUSAL benchmarking dataset

Fig. 2

AUPRC improvement with iterative integration of different predictors. The combined predictors using the max-min prediction score normalization approach were tested on the ALL and the CAUSAL benchmarking datasets, respectively. The predictor integrated at each round of iteration and the AUPRC of the combined predictor are indicated on the line chart. a The AUPRC results of the combined predictors on the ALL benchmarking dataset. b The AUPRC results of the combined predictors on the CAUSAL benchmarking dataset

Fig. 3

The stratified comparison of predictor performance in terms of DSW and MSW. a Dot plots where the AUPRCs of the well-annotated miRNAs (with the top 25% DSW scores) are plotted against AUPRCs of the less-annotated miRNAs (with the last 25% DSW scores). b Dot plots where the AUPRCs of the well-annotated diseases (with the top 25% MSW scores) are plotted against AUPRCs of the less-annotated diseases (with the last 25% DSW scores)

Fig. 4

The comparison of the prediction performance using MISIM 2.0 or MISIM 1.0 miRNA similarity matrix

Fig. 5

The prediction performance for prioritizing disease causal miRNAs. a The ROC curves illustrating the performance in distinguishing causal miRNA-disease associations (as the positive samples) from the non-causal miRNA-disease associations (as the negative samples); only the top ten predictors in terms of AUROC are shown. b–d The violin plots for three predictors that show significant higher prediction scores (via Wilcoxon test) for causal miRNA-disease associations than non-causal miRNA-disease associations