An equivalent relative utility metric for evaluating screening mammography

Abstract

Comparative studies of performance in screening mammography are often ambiguous. A new method will frequently show a higher sensitivity or detection rate than an existing standard with a concomitant increase in false positives or recalls. The authors propose an equivalent relative utility (ERU) metric based on signal detection theory to quantify screening performance in such comparisons. The metric is defined as the relative utility, as defined in classical signal detection theory, needed to make 2 systems equivalent. ERU avoids the problem of requiring a predefined putative relative utility, which has limited application of utility theory in receiver operating characteristic analysis. The metric can be readily estimated from recall and detection rates commonly reported in comparative clinical studies. An important practical advantage of ERU is that in prevalence matched populations, the measure can be estimated without an independent estimate of disease prevalence. Thus estimating ERU does not require a study with long-term follow-up to find cases of missed disease. The approach is applicable to any comparative screening study that reports results in terms of recall and detection rates, although the authors focus exclusively on screening mammography in this work. They derive the ERU from the definition of utility given in classical treatments of signal detection theory. They also investigate reasonable values of relative utility in screening mammography for use in interpreting ERU using data from a large clinical study. As examples of application of ERU, they reanalyze 2 recently published reports using recall and detection rates in screening mammography.

Figure 1. Utility in the ROC domain

A generic ROC curve is shown along with lines of iso-utility as defined in Equation 4. The iso-lines partition the domain into regions of higher utility (upper left) from lower utility (lower right). The relative utility (U_Rel) and the odds ratio for disease prevalence (Q_π) define the slope of iso-utility lines. The optimal operating point on the ROC curve is tangent to an iso-utility line.

Figure 2. Interpretation of Equivalent Relative Utility (ERU)

This schematic figure shows the operating points of two hypothetical screening modalities (M1 and M2) in terms of true-positive and false-positive rates. Lines represent points of equal utility with higher utility to the upper right and lower utility to the lower left. Different lines represent different values of the putative relative utility (U_Rel) according to Equation 4. When U_Rel is equal to the ERU, the iso-utility line passes through both operating points. If U_Rel is less than the ERU, Modality 2 falls on the lower utility side of the iso-line and Modality 1 is superior. Conversely, if U_Rel is greater than the ERU, Modality 2 is superior.

Figure 3. Utility Analysis of the Barlow et al. Mammography Data

ROC curves fit to the aggregate data (A) for a 7-point scale (Global Fit) or a reduced 5-point scale (Local Fit) have similar A_z values (0.920 global and 0.916 local). The relative utility (B) over a limited range of false-positive rates is computed from the slope of the ROC curves using a prevalence of 0.512%. Relative utilities near the 10% False-positive rate bracket the Wagner et al.[12] estimate of 150.

Figure 4. Equivalent Relative Utility Analysis of the Gur et al. Mammography Data

Recall and detection rates are plotted along with the best fit line from the Gur et al. publication, yielding an ERU of 46. We also plot a line with a slope corresponding to an ERU of 150 and least squares fitted intercept.