Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2009 Oct 19:339:b4012.
doi: 10.1136/bmj.b4012.

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study

Lisa Hartling  1 , Maria Ospina, Yuanyuan Liang, Donna M Dryden, Nicola Hooton, Jennifer Krebs Seida, Terry P Klassen
Affiliations
Comparative Study

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study

Lisa Hartling et al. BMJ. .

Abstract

Objectives: To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates.

Design: Cross sectional study. Study sample 163 trials in children.

Main outcome measures: Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted kappa), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall's tau statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression).

Results: Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (kappa=0.13) to substantial (kappa=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8.8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23).

Conclusions: Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Figures

None
Effect size estimates according to risk of bias
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Agency for Healthcare Research and Quality. Systems to rate the strength of scientific evidence. Evidence report/technology assessment No 47. 2002. www.ahrq.gov/clinic/epcsums/strengthsum.htm.
    1. Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA. The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol 2001;54:651-4. - PubMed
    1. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ 1988;138:697-703. - PMC - PubMed
    1. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions version 5.0.0. Cochrane Collaboration, 2008.
    1. Jüni P, Altman DG, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001;323:42-6. - PMC - PubMed