Quasi-experimental study designs series-paper 6: risk of bias assessment

J Clin Epidemiol. 2017 Sep:89:43-52. doi: 10.1016/j.jclinepi.2017.02.015. Epub 2017 Mar 27.


Objectives: Rigorous and transparent bias assessment is a core component of high-quality systematic reviews. We assess modifications to existing risk of bias approaches to incorporate rigorous quasi-experimental approaches with selection on unobservables. These are nonrandomized studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments, and regression-discontinuity designs.

Study design and setting: We review existing risk of bias tools. Drawing on these tools, we present domains of bias and suggest directions for evaluation questions.

Results: The review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of these designs. The paper then presents an approach to evaluating the internal validity of quasi-experiments with selection on unobservables.

Conclusion: We conclude that tools for nonrandomized studies of interventions need to be further developed to incorporate evaluation questions for quasi-experiments with selection on unobservables.

Keywords: Difference in differences; Instrumental variables; Interrupted time series; Meta-Analysis; Natural experiment; Quasi-experiment; Regression discontinuity; Risk of bias; Systematic review.

Publication types

  • Review
  • Systematic Review

MeSH terms

  • Bias*
  • Humans
  • Non-Randomized Controlled Trials as Topic / statistics & numerical data*
  • Research Design
  • Risk Assessment