Application of threshold-bias independent analysis to eye-tracking and FROC data

Abstract

Rationale and objectives: Studies of medical image interpretation have focused on either assessing radiologists' performance using, for example, the receiver operating characteristic (ROC) paradigm, or assessing the interpretive process by analyzing their eye-tracking (ET) data. Analysis of ET data has not benefited from threshold-bias independent figures of merit (FOMs) analogous to the area under the receiver operating characteristic (ROC) curve. The aim was to demonstrate the feasibility of such FOMs and to measure the agreement between FOMs derived from free-response ROC (FROC) and ET data.

Methods: Eight expert breast radiologists interpreted a case set of 120 two-view mammograms while eye-position data and FROC data were continuously collected during the interpretation interval. Regions that attract prolonged (>800 ms) visual attention were considered to be virtual marks, and ratings based on the dwell and approach-rate (inverse of time-to-hit) were assigned to them. The virtual ratings were used to define threshold-bias independent FOMs in a manner analogous to the area under the trapezoidal alternative FROC (AFROC) curve (0 = worst, 1 = best). Agreement at the case level (0.5 = chance, 1 = perfect) was measured using the jackknife and 95% confidence intervals (CI) for the FOMs and agreement were estimated using the bootstrap.

Results: The AFROC mark-ratings' FOM was largest at 0.734 (CI 0.65-0.81) followed by the dwell at 0.460 (0.34-0.59) and then by the approach-rate FOM 0.336 (0.25-0.46). The differences between the FROC mark-ratings' FOM and the perceptual FOMs were significant (P < .05). All pairwise agreements were significantly better then chance: ratings vs. dwell 0.707 (0.63-0.88), dwell vs. approach-rate 0.703 (0.60-0.79) and rating vs. approach-rate 0.606 (0.53-0.68). The ratings vs. approach-rate agreement was significantly smaller than the dwell vs. approach-rate agreement (P = .008).

Conclusions: Leveraging current methods developed for analyzing observer performance data could complement current ways of analyzing ET data and lead to new insights.

Fig. 1

Schematic of the data collection and processing to obtain real and virtual marks: the radiologists interpreted the images using a two-monitor workstation. Concurrently, and for the duration of the interpretation, an ASL eye-position tracking system determined the line-of-gaze. The ASL fixation and clustering algorithms are described in the text. The proximity criterion, defined as 2.5° of visual angle, is the maximum distance between a lesion center and a mark for the mark to be considered a LL (correct localization). Non-lesion localizations are all other marks. ASL = Applied Sciences Laboratory; NL = non-lesion localization; LL = lesion localization.

Fig. 2

Example of (a) small-clusters in yellow, (b) big-clusters, prior to threshold, in green and (c) big-clusters, after applying the 800 ms threshold, in blue. The green ‘diamond’ symbol marks the location where search started, whereas the red ‘cross’ symbol indicates the locations marked by the radiologist as containing a malignant lesion. The small yellow dots mark the raw eye-position data. The red circles mark the true locations of the lesions in the two views.

Fig. 2

Example of (a) small-clusters in yellow, (b) big-clusters, prior to threshold, in green and (c) big-clusters, after applying the 800 ms threshold, in blue. The green ‘diamond’ symbol marks the location where search started, whereas the red ‘cross’ symbol indicates the locations marked by the radiologist as containing a malignant lesion. The small yellow dots mark the raw eye-position data. The red circles mark the true locations of the lesions in the two views.

Fig. 2

Example of (a) small-clusters in yellow, (b) big-clusters, prior to threshold, in green and (c) big-clusters, after applying the 800 ms threshold, in blue. The green ‘diamond’ symbol marks the location where search started, whereas the red ‘cross’ symbol indicates the locations marked by the radiologist as containing a malignant lesion. The small yellow dots mark the raw eye-position data. The red circles mark the true locations of the lesions in the two views.