Influence of trial sample size on treatment effect estimates: meta-epidemiological study

Abstract

Objective: To assess the influence of trial sample size on treatment effect estimates within meta-analyses.

Design: Meta-epidemiological study.

Data sources: 93 meta-analyses (735 randomised controlled trials) assessing therapeutic interventions with binary outcomes, published in the 10 leading journals of each medical subject category of the Journal Citation Reports or in the Cochrane Database of Systematic Reviews.

Data extraction: Sample size, outcome data, and risk of bias extracted from each trial.

Data synthesis: Trials within each meta-analysis were sorted by their sample size: using quarters within each meta-analysis (from quarter 1 with 25% of the smallest trials, to quarter 4 with 25% of the largest trials), and using size groups across meta-analyses (ranging from <50 to ≥ 1000 patients). Treatment effects were compared within each meta-analysis between quarters or between size groups by average ratios of odds ratios (where a ratio of odds ratios less than 1 indicates larger effects in smaller trials).

Results: Treatment effect estimates were significantly larger in smaller trials, regardless of sample size. Compared with quarter 4 (which included the largest trials), treatment effects were, on average, 32% larger in trials in quarter 1 (which included the smallest trials; ratio of odds ratios 0.68, 95% confidence interval 0.57 to 0.82), 17% larger in trials in quarter 2 (0.83, 0.75 to 0.91), and 12% larger in trials in quarter 3 (0.88, 0.82 to 0.95). Similar results were obtained when comparing treatment effect estimates between different size groups. Compared with trials of 1000 patients or more, treatment effects were, on average, 48% larger in trials with fewer than 50 patients (0.52, 0.41 to 0.66) and 10% larger in trials with 500-999 patients (0.90, 0.82 to 1.00).

Conclusions: Treatment effect estimates differed within meta-analyses solely based on trial sample size, with stronger effect estimates seen in small to moderately sized trials than in the largest trials.

Fig 1 Comparison of treatment effect estimates between trial sample sizes, grouped by quarters (from quarter 1 with the smallest trials, to quarter 4 with the largest trials) and by size groups (from <50 patients to ≥1000 patients). Both analyses included all 93 individual meta-analyses (735 randomised controlled trials)

Fig 2 Comparison of treatment effect estimates between trial sample sizes grouped by quarters (from quarter 1 with the smallest trials, to quarter 4 with the largest trials) and by fixed thresholds (according to numbers of patients). All 93 meta-analyses did not contribute to the analysis, depending on the threshold used (at least one trial with a sample size less and more than the threshold within each meta-analysis was required to perform these analyses). *Five meta-analyses did not contribute to the analysis (three included fewer than four trials; for two trials, the meta-regression model did not converge, therefore, the ratio of odds ratios could not be obtained). †One meta-analysis did not contribute to the analysis (the meta-regression model did not converge, therefore, the ratio of odds ratios could not be obtained)