Benchmarks for detecting 'breakthroughs' in clinical trials: empirical assessment of the probability of large treatment effects using kernel density estimation

BMJ Open. 2014 Oct 21;4(10):e005249. doi: 10.1136/bmjopen-2014-005249.


Objective: To understand how often 'breakthroughs,' that is, treatments that significantly improve health outcomes, can be developed.

Design: We applied weighted adaptive kernel density estimation to construct the probability density function for observed treatment effects from five publicly funded cohorts and one privately funded group.

Data sources: 820 trials involving 1064 comparisons and enrolling 331,004 patients were conducted by five publicly funded cooperative groups. 40 cancer trials involving 50 comparisons and enrolling a total of 19,889 patients were conducted by GlaxoSmithKline.

Results: We calculated that the probability of detecting treatment with large effects is 10% (5-25%), and that the probability of detecting treatment with very large treatment effects is 2% (0.3-10%). Researchers themselves judged that they discovered a new, breakthrough intervention in 16% of trials.

Conclusions: We propose these figures as the benchmarks against which future development of 'breakthrough' treatments should be measured.


Publication types

  • Research Support, N.I.H., Extramural

MeSH terms

  • Benchmarking*
  • Clinical Trials as Topic*
  • Humans
  • Neoplasms / therapy*
  • Probability
  • Statistics as Topic*
  • Treatment Outcome