The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models

Abstract

Background: Many measures of prediction accuracy have been developed. However, the most popular ones in typical medical outcome prediction settings require additional investigation of calibration.

Methods: We show how rescaling the Brier score produces a measure that combines discrimination and calibration in one value and improves interpretability by adjusting for a benchmark model. We have called this measure the index of prediction accuracy (IPA). The IPA permits a common interpretation across binary, time to event, and competing risk outcomes. We illustrate this measure using example datasets.

Results: The IPA is simple to compute, and example code is provided. The values of the IPA appear very interpretable.

Conclusions: IPA should be a prominent measure reported in studies of medical prediction model performance. However, IPA is only a measure of average performance and, by default, does not measure the utility of a medical decision.

Keywords: Accuracy; Brier score; Prediction.

Fig. 1

Performance metric as a function of event prevalence. Legend: the solid line is the Brier score of the model which predicts prevalence to all subjects. The dashed line is the corresponding root Brier score

Fig. 2

Illustration of IPA as a function of discrimination, without and with miscalibration

Fig. 3

Absolute risk of progression accounting for non-cancer death as a competing risk

Fig. 4

Illustration for the effect on IPA from changing the prediction horizon

Fig. 5

Comparison of rival prediction models