Problems caused by heterogeneity in meta-analysis: a case study of acupuncture trials

Acupunct Med. 2014 Feb;32(1):56-61. doi: 10.1136/acupmed-2013-010364. Epub 2013 Oct 18.


Objectives: To illustrate the pitfalls of using meta-analysis to combine estimates of effect in trials that are highly varied and have a high potential for bias.

Methods: We used a random-effects meta-analysis to pool the results of 51 sham-controlled acupuncture trials of chronic pain published in English before 2008 and explored the heterogeneity using meta-regression. We repeated the process on a subset of these trials that used a visually credible non-penetrating sham device as control (N = 12).

Results: In both analyses there were high levels of heterogeneity and many studies were at risk from potential bias. The heterogeneity was not explained by meta-regression.

Conclusions: Trials of interventions that have high potential for bias, such as many in the acupuncture literature, do not meet the assumptions of the statistical procedure that underlie random-effects meta-analysis. Even in the absence of bias, heterogeneity in meta-analyses is not accounted for by the CIs around the pooled estimate.

Keywords: Acupuncture; Statistics & Research Methods; Systematic Reviews.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Acupuncture Therapy*
  • Humans
  • Meta-Analysis as Topic*
  • Randomized Controlled Trials as Topic
  • Regression Analysis
  • Research Design / standards*