Nonrandomized studies are not always found even when selection criteria for health systems intervention reviews include them: a methodological study

Abstract

Objective: Systematic reviews within the Cochrane Effective Practice and Organisation of Care Group (EPOC) can include both randomized and nonrandomized study designs. We explored how many EPOC reviews consider and identify nonrandomized studies, and whether the proportion of nonrandomized studies identified is linked to the review topic.

Study design and setting: We recorded the study designs considered in 65 EPOC reviews. For reviews that considered nonrandomized studies, we calculated the proportion of identified studies that were nonrandomized and explored whether there were differences in the proportion of nonrandomized studies according to the review topic.

Results: Fifty-one (78.5%) reviews considered nonrandomized studies. Forty-six of these reviews found nonrandomized studies, but the proportion varied a great deal (median, 33%; interquartile range, 25--50%). Reviews of health care delivery interventions had lower proportions of nonrandomized studies than those of financial and governance interventions.

Conclusion: Most EPOC reviews consider nonrandomized studies, but the degree to which they find them varies. As nonrandomized studies are believed to be at higher risk of bias and their inclusion entails a considerable effort, review authors should consider whether the benefits justify the inclusion of these designs. Research should explore whether it is more useful to consider nonrandomized studies in reviews of some intervention types than others.