Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 14, 120

Thresholds for Statistical and Clinical Significance in Systematic Reviews With Meta-Analytic Methods


Thresholds for Statistical and Clinical Significance in Systematic Reviews With Meta-Analytic Methods

Janus Christian Jakobsen et al. BMC Med Res Methodol.


Background: Thresholds for statistical significance when assessing meta-analysis results are being insufficiently demonstrated by traditional 95% confidence intervals and P-values. Assessment of intervention effects in systematic reviews with meta-analysis deserves greater rigour.

Methods: Methodologies for assessing statistical and clinical significance of intervention effects in systematic reviews were considered. Balancing simplicity and comprehensiveness, an operational procedure was developed, based mainly on The Cochrane Collaboration methodology and the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidelines.

Results: We propose an eight-step procedure for better validation of meta-analytic results in systematic reviews (1) Obtain the 95% confidence intervals and the P-values from both fixed-effect and random-effects meta-analyses and report the most conservative results as the main results. (2) Explore the reasons behind substantial statistical heterogeneity using subgroup and sensitivity analyses (see step 6). (3) To take account of problems with multiplicity adjust the thresholds for significance according to the number of primary outcomes. (4) Calculate required information sizes (≈ the a priori required number of participants for a meta-analysis to be conclusive) for all outcomes and analyse each outcome with trial sequential analysis. Report whether the trial sequential monitoring boundaries for benefit, harm, or futility are crossed. (5) Calculate Bayes factors for all primary outcomes. (6) Use subgroup analyses and sensitivity analyses to assess the potential impact of bias on the review results. (7) Assess the risk of publication bias. (8) Assess the clinical significance of the statistically significant review results.

Conclusions: If followed, the proposed eight-step procedure will increase the validity of assessments of intervention effects in systematic reviews of randomised clinical trials.


Figure 1
Figure 1
A figure showing how Bayes factor will change according to different observed effects. The red left vertical line represents the null hypothesis (an effect of null), and the right green vertical line represents an alternative hypothesis to the null hypothesis with an effect of 1.0. The black curve shows that Bayes factor will be 1.0 when the observed effect size is exactly half of the effect size of the alternative hypothesis; and the curve shows that Bayes factor will decrease with increasing observed effect sizes.

Similar articles

See all similar articles

Cited by 102 articles

See all "Cited by" articles


    1. Sun X, Guyatt G. Meta-analysis of randomized trials for health care interventions: one for all? J Evid Based Med. 2009;2(1):53–56. doi: 10.1111/j.1756-5391.2009.01006.x. - DOI - PubMed
    1. Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, Neumann I, Carrasco-Labra A, Agoritsas T, Hatala R, Meade MO, Wyer P, Cook DJ, Guyatt G. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312(2):171–179. doi: 10.1001/jama.2014.5559. - DOI - PubMed
    1. Stegenga J. Is meta-analysis the platinum standard of evidence? Stud Hist Philos Biol Biomed Sci. 2011;42(4):497–507. doi: 10.1016/j.shpsc.2011.07.003. - DOI - PubMed
    1. Hennekens CH, DeMets D. The need for large-scale randomized evidence without undue emphasis on small trials, meta-analyses, or subgroup analyses. JAMA. 2009;302(21):2361–2362. doi: 10.1001/jama.2009.1756. - DOI - PubMed
    1. Borzak S, Ridker PM. Discordance between meta-analyses and large-scale randomized, controlled trials. Examples from the management of acute myocardial infarction. Ann Intern Med. 1995;123(11):873–877. doi: 10.7326/0003-4819-123-11-199512010-00010. - DOI - PubMed
Pre-publication history
    1. The pre-publication history for this paper can be accessed here:

Publication types

MeSH terms

LinkOut - more resources