Synthesis of observational studies should consider credibility ceilings

J Clin Epidemiol. 2009 Feb;62(2):115-22. doi: 10.1016/j.jclinepi.2008.05.014.


Objective: Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.

Study design and setting: We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).

Results: Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.

Conclusions: Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.

MeSH terms

  • Algorithms
  • Bias*
  • Data Interpretation, Statistical*
  • Fatty Acids, Omega-3 / administration & dosage
  • Hair Dyes / adverse effects
  • Hospital Mortality
  • Hospitals, Teaching / statistics & numerical data
  • Humans
  • Likelihood Functions
  • Lymphoma, Non-Hodgkin / epidemiology
  • Meta-Analysis as Topic*
  • Research Design / standards
  • Risk


  • Fatty Acids, Omega-3
  • Hair Dyes