Avoidable waste of research related to inadequate methods in clinical trials

Abstract

Objective: To assess the waste of research related to inadequate methods in trials included in Cochrane reviews and to examine to what extent this waste could be avoided. A secondary objective was to perform a simulation study to re-estimate this avoidable waste if all trials were adequately reported.

Design: Methodological review and simulation study.

Data sources: Trials included in the meta-analysis of the primary outcome of Cochrane reviews published between April 2012 and March 2013.

Data extraction and synthesis: We collected the risk of bias assessment made by the review authors for each trial. For a random sample of 200 trials with at least one domain at high risk of bias, we re-assessed risk of bias and identified all related methodological problems. For each problem, possible adjustments were proposed that were then validated by an expert panel also evaluating their feasibility (easy or not) and cost. Avoidable waste was defined as trials with at least one domain at high risk of bias for which easy adjustments with no or minor cost could change all domains to low risk. In the simulation study, after extrapolating our re-assessment of risk of bias to all trials, we considered each domain rated as unclear risk of bias as missing data and used multiple imputations to determine whether they were at high or low risk.

Results: Of 1286 trials from 205 meta-analyses, 556 (43%) had at least one domain at high risk of bias. Among the sample of 200 of these trials, 142 were confirmed as high risk; in these, we identified 25 types of methodological problem. Adjustments were possible in 136 trials (96%). Easy adjustments with no or minor cost could be applied in 71 trials (50%), resulting in 17 trials (12%) changing to low risk for all domains. So the avoidable waste represented 12% (95% CI 7% to 18%) of trials with at least one domain at high risk. After correcting for incomplete reporting, avoidable waste due to inadequate methods was estimated at 42% (95% CI 36% to 49%).

Conclusions: An important burden of wasted research is related to inadequate methods. This waste could be partly avoided by simple and inexpensive adjustments.

Fig 1 Flow diagram of the selection of trials

Fig 2 Avoidable waste of research related to inadequate methods in trials with at least one domain confirmed at high risk of bias (n=142). This figure summarises the evolution in risk of bias assessment per domain for each trial, initially (I), after easy (E) methodological adjustments, or after all (A) possible methodological adjustments. Each spoke represents one of the 142 trials with at least one domain at high risk of bias. The bricks are a visual representation of the risk of bias assessment per domain, red indicating high risk of bias; yellow, unclear risk; and green, low risk for each trial. Every concentric circle represents one of the key domains of the risk of bias tool with “sequence generation” being the furthest circle from the centre, and “incomplete outcome data”, the central circle. The most external circle represents a quick overview of the risk of bias for each trial (red if at least one domain is at high risk, yellow if at least one domain is at unclear risk, and green if all domains are at low risk). The histogram is a representation of the risk of bias, per domain, across the 142 high risk trials, initially (I), after easy (E) methodological adjustments, or after all (A) possible methodological adjustments.