Objective: Meta-analysis (MA) is the application of quantitative techniques for the purposes of summarizing data from individual studies. This type of review has many advantages over traditional reviews. However, different investigators performing MAs on the same data set have reached different conclusions. These reliability problems have been attributed to differences in the quality of the implemented meta-analytic procedures. We, therefore, examined the chronic pain treatment meta-analytic literature for MA procedure quality and for the consistency of conclusions. DESIGN, SETTING, PARTICIPANTS, OUTCOME MEASURES: Chronic pain treatment MAs were isolated according to inclusion/exclusion criteria. Data from these MAs were abstracted into structured tables. Table format reflected eight meta-analytic procedures identified previously as being important to MA implementation quality. These were: adequacy of retrieval, publication bias, inclusion/exclusion criteria, abstraction of data, quality, homogeneity/heterogeneity, independence, and statistical versus clinical interpretation. Each meta-analytic procedure was then independently rated by two raters. Rating results were then analyzed by procedure for each individual MA for percentage scores out of 100%, and mean scores. For MAs addressing the same topic area (pain facility treatment, antidepressant treatment, manipulation treatment) direction of effect size was noted. Mean effect sizes were calculated for these subgroups.
Results: Sixteen chronic pain treatment MAs fulfilled inclusion/exclusion criteria. Mean procedure ratings indicated that four procedures may not be implemented adequately. These were publication bias, abstraction of data, quality, and homogeneity/heterogeneity. There was wide MA implementation score variability, with 37.5% scoring less than 50%. The effect sizes of the MA subgroups demonstrated replicate nonvariability.
Conclusions: Some meta-analytic procedures could be interpreted to be implemented inadequately in some chronic pain treatment MAs. There is wide variability between individual chronic pain treatment MAs on adequacy of implementation of these procedures. However, the effect sizes of the different MA subgroups demonstrated consistency. This finding indicates that for these MA subgroups, MA results are consistent between authors. In addition, chronic pain MAs, as compared with other groups of MAs, appear to address some of the procedures in a more adequate fashion. Future chronic pain MAs should concentrate on improving the quality of their methods with particular emphasis on the above four procedures. Because of potential validity problems with these results, these data cannot and should not be used to make administrative decisions about previous MAs.