The F score ranks diagnostic tests and prediction models inconsistently with their clinical utility

Abstract

Background: The F score is derived from precision (positive predictive value) and recall (sensitivity). It is increasingly used to evaluate diagnostic tests and prediction models in the machine learning literature. Although precision and recall can be differentially weighted using a parameter β, almost all applications use equal weighting, with β set to 1.

Methods: We considered a cancer detection scenario to explore the properties of the F score in comparison to net benefit, a well-established method for evaluating the clinical utility of tests and models. Because missing cancer can be fatal and biopsies are an invasive procedure, we would favor a test with high sensitivity.

Results: F scores did not provide a rank ordering of tests consistent with utility. F1 was highest for a test with greater specificity; in contrast, the conventional decision analytic measure, net benefit, rank ordered tests consistent with clinical intuition, with the highest sensitivity test favored. While it might be argued that F scores can be made consistent with utility by choosing a value of β different from 1, we found it is impossible to rationally prespecify β for any given clinical scenario, as even small changes in prevalence led to undesirable rank orderings for a given β being inconsistent with utility.

Conclusion: The F score ranks diagnostic tests and prediction models inconsistently with their clinical utility. Moreover, the F score does not have an interpretable unit, does not allow for a comparison with a strategy assuming all are negative, and requires dichotomization of models. In contrast, standard decision-analytic measures - net benefit and decision curve analysis - allow rational and consistent choice of weighting, have an interpretable unit, can evaluate the strategy of assuming all are negative, and do not require dichotomization of continuous models. Consistent with TRIPOD AI we recommend that net benefit, alongside discrimination and calibration, be used for the evaluation of diagnostic tests and prediction models.

Keywords: Diagnostic tests; F score; Net benefit; Prediction models; Utility.