Susceptibility to fraud in systematic reviews: lessons from the Reuben case

Anesthesiology. 2009 Dec;111(6):1279-89. doi: 10.1097/ALN.0b013e3181c14c3d.

Abstract

Background: Dr. Scott Reuben allegedly fabricated data. The authors of the current article examined the impact of Reuben reports on conclusions of systematic reviews.

Methods: The authors searched in ISI Web of Knowledge systematic reviews citing Reuben reports. Systematic reviews were grouped into one of three categories: I, only cited but did not include Reuben reports; II, retrieved and considered, but eventually excluded Reuben reports; III, included Reuben reports. For quantitative systematic reviews (i.e., meta-analyses), a relevant difference was defined as a significant result becoming nonsignificant (or vice versa) by excluding Reuben reports. For qualitative systematic reviews, each author decided independently whether noninclusion of Reuben reports would have changed conclusions.

Results: Twenty-five systematic reviews (5 category I, 6 category II, 14 category III) cited 27 Reuben reports (published 1994-2007). Most tested analgesics in surgical patients. One of 6 quantitative category III reviews would have reached different conclusions without Reuben reports. In all 6 (30 subgroup analyses involving Reuben reports), exclusion of Reuben reports never made any difference when the number of patients from Reuben reports was less than 30% of all patients included in the analysis. Of 8 qualitative category III reviews, all authors agreed that one would certainly have reached different conclusions without Reuben reports. For another 4, the authors' judgment was not unanimous.

Conclusions: Carefully performed systematic reviews proved robust against the impact of Reuben reports. Quantitative systematic reviews were vulnerable if the fraudulent data were more than 30% of the total. Qualitative systematic reviews seemed at greater risk than quantitative.

MeSH terms

  • Analgesics / adverse effects
  • Analgesics / therapeutic use
  • Cyclooxygenase 2 Inhibitors / adverse effects
  • Cyclooxygenase 2 Inhibitors / therapeutic use
  • Humans
  • Meta-Analysis as Topic*
  • Pain, Postoperative / drug therapy
  • Randomized Controlled Trials as Topic
  • Review Literature as Topic*
  • Scientific Misconduct*
  • Treatment Outcome

Substances

  • Analgesics
  • Cyclooxygenase 2 Inhibitors