GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

J Clin Epidemiol. 2019 Jul;111:105-114. doi: 10.1016/j.jclinepi.2018.01.012. Epub 2018 Feb 9.


Objective: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

Study design and setting: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

Results: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

Conclusions: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Keywords: Certainty of the evidence; GRADE; Nonrandomized studies; Quality of evidence; ROBINS; Risk of bias.

MeSH terms

  • Bias*
  • Clinical Trials as Topic / standards
  • Evidence-Based Medicine / methods
  • Evidence-Based Medicine / standards*
  • Humans
  • Observational Studies as Topic / standards
  • Practice Guidelines as Topic* / standards
  • Risk Assessment
  • Risk Factors
  • Systematic Reviews as Topic
  • Uncertainty