Agreement studies in radiology research

Abstract

Purpose: The goal of this study was to estimate the frequency and the quality of agreement studies published in diagnostic imaging journals.

Materials and methods: All studies published between January 2011 and December 2012 in four radiology journals were reviewed. Four trained readers evaluated agreement studies using a 24-item form that included the 15 items of the Guidelines for Reporting Reliability and Agreement Studies criteria.

Results: Of 2229 source titles, 280 studies (13%) reported agreement. The mean number of patients per study was 81±99 (SD) (range, 0-180). Justification for sample size was found in 9 studies (3%). The number of raters was≤2 in 226 studies (81%). No intra-observer study was performed in 212 (76%) articles. Confidence intervals and interpretation of statistical estimates were provided in 98 (35%) and 147 (53%) of the studies, respectively. In 168 studies (60%), the agreement study was not mentioned in the discussion section. In 8 studies (3%), reporting of the agreement study was judged to be adequate. Twenty studies (7%) were dedicated to agreement.

Conclusion: Agreement studies are preliminary and not adequately reported. Studies dedicated to agreement are infrequent. They are research opportunities that should be promoted.

Keywords: Agreement; Inter-observer; Intra-observer; Reliability; Reproducibility.