Reducing the number of reader interpretations in MRMC studies

Acad Radiol. 2009 Feb;16(2):209-17. doi: 10.1016/j.acra.2008.05.014.


Rationale and objectives: Multireader, multicase (MRMC) receiver-operating characteristic studies often require large numbers of patients, readers, and reader interpretations. The objective of this work is to evaluate a new "mixed" MRMC study design that reduces the number of reader interpretations.

Materials and methods: As compared to the traditional MRMC design, the number of reader interpretations and the number of cases that must be truth-verified for the new mixed design was evaluated theoretically and empirically for various correlation values and sample sizes.

Results: For large MRMC studies, the new mixed design offers a substantial savings in the number of reader interpretations if the magnitude of the difference in between-reader correlations is not zero. For example, compared to a traditional design with 20 readers, 200 total cases, and a difference in between-reader correlations of 0.05, the newly proposed mixed design requires each reader to interpret only 132 cases, but at a cost of truth-verifying an additional 64 cases.

Conclusions: The mixed design can reduce the number of cases that readers need to interpret and the overall duration of a study, but at a cost in terms of the number of cases that must be truth-verified. The mixed design is particularly useful for studies where the condition being detected is not rare and patients routinely undergo the gold standard assessment.

Publication types

  • Evaluation Study

MeSH terms

  • Algorithms*
  • Data Interpretation, Statistical*
  • Image Enhancement / methods
  • Image Interpretation, Computer-Assisted / methods*
  • Observer Variation*
  • ROC Curve*
  • Reproducibility of Results
  • Sample Size
  • Sensitivity and Specificity