Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

doi:10.1016/j.jclinepi.2022.08.015

. 2022 Dec:152:300-306.

doi: 10.1016/j.jclinepi.2022.08.015. Epub 2022 Sep 19.

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

Affiliations

¹ Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA. Electronic address: ian_saldanha@brown.edu.
² Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA.
³ Evidence-Based Practice Center Program, Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, MD, USA.
⁴ Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA.
⁵ Cerner Enviza, Inc., Kansas City, MO, USA.
⁶ Comparative Health Outcomes, Policy, and Economics (CHOICE) Institute, University of Washington School of Pharmacy, Seattle, WA, USA.
⁷ Department of Health Services Research, Management, and Policy, University of Florida, Gainesville, FL, USA.
⁸ Oregon Health and Science University, Portland, OR, USA.
⁹ Pacific Northwest Evidence-Based Practice Center, Portland, OR, USA; Aggregate Analytics, Inc., Fircrest, WA, USA.
¹⁰ Department of Medical and Clinical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR, USA.
¹¹ Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA.
¹² Minneapolis VA Center for Care Delivery and Outcomes Research, University of Minnesota Schools of Medicine and Public Health, Minneapolis, MN, USA.
¹³ RTI International, Research Triangle Park, NC, USA.

PMID: 36245131
PMCID: PMC10777810
DOI: 10.1016/j.jclinepi.2022.08.015

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

Ian J Saldanha et al. J Clin Epidemiol. 2022 Dec.

. 2022 Dec:152:300-306.

doi: 10.1016/j.jclinepi.2022.08.015. Epub 2022 Sep 19.

Authors

Affiliations

¹ Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA. Electronic address: ian_saldanha@brown.edu.
² Center for Evidence Synthesis in Health, Department of Health Services, Policy, and Practice, Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA.
³ Evidence-Based Practice Center Program, Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, MD, USA.
⁴ Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA.
⁵ Cerner Enviza, Inc., Kansas City, MO, USA.
⁶ Comparative Health Outcomes, Policy, and Economics (CHOICE) Institute, University of Washington School of Pharmacy, Seattle, WA, USA.
⁷ Department of Health Services Research, Management, and Policy, University of Florida, Gainesville, FL, USA.
⁸ Oregon Health and Science University, Portland, OR, USA.
⁹ Pacific Northwest Evidence-Based Practice Center, Portland, OR, USA; Aggregate Analytics, Inc., Fircrest, WA, USA.
¹⁰ Department of Medical and Clinical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR, USA.
¹¹ Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA.
¹² Minneapolis VA Center for Care Delivery and Outcomes Research, University of Minnesota Schools of Medicine and Public Health, Minneapolis, MN, USA.
¹³ RTI International, Research Triangle Park, NC, USA.

PMID: 36245131
PMCID: PMC10777810
DOI: 10.1016/j.jclinepi.2022.08.015

Abstract

Objectives: We developed guidance to inform decisions regarding the inclusion of nonrandomized studies of interventions (NRSIs) in systematic reviews (SRs) of the effects of interventions.

Study design and setting: The guidance workgroup comprised SR experts and used an informal consensus generation method.

Results: Instead of recommending NRSI inclusion only if randomized controlled trials (RCTs) are insufficient to address the SR key question, different topics may require different decisions regarding NRSI inclusion. We identified important considerations to inform such decisions from topic refinement through protocol development. During topic scoping and refinement, considerations were related to the clinical decisional dilemma, adequacy of RCTs to address the key questions, risk of bias in NRSIs, and the extent to which NRSIs are likely to complement RCTs. When NRSIs are included, during SR team formation, familiarity with topic-specific data sources and advanced analytic methods for NRSIs should be considered. During protocol development, the decision regarding NRSI inclusion or exclusion should be justified, and potential implications explained. When NRSIs are included, the protocol should describe the processes for synthesizing evidence from RCTs and NRSIs and determining the overall strength of evidence.

Conclusion: We identified specific considerations for decisions regarding NRSI inclusion in SRs and highlight the importance of flexibility and transparency.

Keywords: Effectiveness; Guidance; Interventions; Nonrandomized studies; Study eligibility; Systematic reviews.

PubMed Disclaimer

Conflict of interest statement

Conflicts of interest: The authors have no conflicts of interest to disclose.

Disclosure: The authors have no relevant financial relationships to disclose.

Cited by

The reporting quality of meta-epidemiological studies needs substantial improvement: a research on research study.
Long Y, Zheng Y, Wang X, Guo Q, Zhang N, Deng Y, Tang R, Li Z, Du L. Long Y, et al. Syst Rev. 2024 Sep 28;13(1):244. doi: 10.1186/s13643-024-02661-7. Syst Rev. 2024. PMID: 39342302 Free PMC article.
Integrating randomized controlled trials and non-randomized studies of interventions to assess the effect of rare events: a Bayesian re-analysis of two meta-analyses.
Zhou Y, Yao M, Mei F, Ma Y, Huan J, Zou K, Li L, Sun X. Zhou Y, et al. BMC Med Res Methodol. 2024 Sep 27;24(1):219. doi: 10.1186/s12874-024-02347-7. BMC Med Res Methodol. 2024. PMID: 39333867 Free PMC article.
Impact of Cognitive Behavior Therapy on Sexual Dysfunction of Women in Reproductive Age: A Systematic Review.
Sharifipour F, Qaderi K, Peighambardoost R, Zahedian M, Azizi F, Faal Siahkal S. Sharifipour F, et al. Int J Sex Health. 2024 Jun 3;36(3):287-301. doi: 10.1080/19317611.2024.2360731. eCollection 2024. Int J Sex Health. 2024. PMID: 39148917
Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study.
Yao M, Wang Y, Busse JW, Briel M, Mei F, Li G, Zou K, Li L, Sun X. Yao M, et al. BMJ Open. 2023 Jul 26;13(7):e073232. doi: 10.1136/bmjopen-2023-073232. BMJ Open. 2023. PMID: 37495391 Free PMC article. Clinical Trial.

References

1. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:1–173. iii-x. - PubMed
1. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology 2004;15:615–25. - PubMed
1. Sterne JA, Hernán MA, Reeves BC, Savovic J, Berkman ND, Viswanathan V, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. - PMC - PubMed
1. Saldanha IJ, Skelly AC, Ley KV, Wang Z, Berliner E, Bass EB, et al. Inclusion of non-randomized studies of interventions in systematic reviews of intervention effectiveness: an update. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022. - PubMed
1. Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions-a taxonomy without labels. J Clin Epidemiol 2017;89:30–42. - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Research Materials
- NCI CPTC Antibody Characterization Program

[1] Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:1–173. iii-x. - PubMed

[2] Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:1–173. iii-x. - PubMed

[3] Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology 2004;15:615–25. - PubMed

[4] Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology 2004;15:615–25. - PubMed

[5] Sterne JA, Hernán MA, Reeves BC, Savovic J, Berkman ND, Viswanathan V, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. - PMC - PubMed

[6] Sterne JA, Hernán MA, Reeves BC, Savovic J, Berkman ND, Viswanathan V, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. - PMC - PubMed

[7] Saldanha IJ, Skelly AC, Ley KV, Wang Z, Berliner E, Bass EB, et al. Inclusion of non-randomized studies of interventions in systematic reviews of intervention effectiveness: an update. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022. - PubMed

[8] Saldanha IJ, Skelly AC, Ley KV, Wang Z, Berliner E, Bass EB, et al. Inclusion of non-randomized studies of interventions in systematic reviews of intervention effectiveness: an update. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022. - PubMed

[9] Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions-a taxonomy without labels. J Clin Epidemiol 2017;89:30–42. - PMC - PubMed

[10] Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions-a taxonomy without labels. J Clin Epidemiol 2017;89:30–42. - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

Affiliations

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

Authors

Affiliations

Abstract

Conflict of interest statement

Similar articles

Cited by

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Research Materials