Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 15, 480

In Search of Justification for the Unpredictability Paradox


In Search of Justification for the Unpredictability Paradox

Jeremy Howick et al. Trials.


A 2011 Cochrane Review found that adequately randomized trials sometimes revealed larger, sometimes smaller, and often similar effect sizes to inadequately randomized trials. However, they found no average statistically significant difference in effect sizes between the two study types. Yet instead of concluding that adequate randomization had no effect the review authors postulated the "unpredictability paradox", which states that randomized and non-randomized studies differ, but in an unpredictable direction. However, stipulating the unpredictability paradox is problematic for several reasons: 1) it makes the authors' conclusion that adequate randomization makes a difference unfalsifiable—if it turned out that adequately randomized trials had significantly different average results from inadequately randomized trials the authors could have pooled the results and concluded that adequate randomization protected against bias; 2) it leaves other authors of reviews with similar results confused about whether or not to pool results (and hence which conclusions to draw); 3) it discourages researchers from investigating the conditions under which adequate randomization over- or under-exaggerates apparent treatment benefits; and 4) it could obscure the relative importance of allocation concealment and blinding which may be more important than adequate randomization.


Figure 1
Figure 1
Pooled results from adequately and inadequately randomized trials in the Odgaard-Jensen and colleagues Cochrane Review [5]. CI, confidence interval; IV, independent variable; RCT, randomized controlled trial; SD, standard deviation; Std, standardized.

Similar articles

See all similar articles


    1. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408–412. doi: 10.1001/jama.1995.03520290060030. - DOI - PubMed
    1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352:609–613. doi: 10.1016/S0140-6736(98)01085-X. - DOI - PubMed
    1. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Inter Med. 2001;135:982–989. doi: 10.7326/0003-4819-135-11-200112040-00010. - DOI - PubMed
    1. Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323:42–46. doi: 10.1136/bmj.323.7303.42. - DOI - PMC - PubMed
    1. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD. Randomization to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. 2011;4:MR000012. - PubMed

Publication types

MeSH terms

LinkOut - more resources