Generalizability of the findings from a randomized controlled trial of a web-based substance use disorder intervention

Am J Addict. 2018 Apr;27(3):231-237. doi: 10.1111/ajad.12714. Epub 2018 Mar 23.


Background and objectives: There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population.

Methods: The sample of the web-based SUD intervention (Therapeutic Education System vs. Treatment-as-usual; n = 507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, we computed weighted treatment effects on retention and abstinence.

Results: Substantial differences between the RCT sample and the target population was demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p < .001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight.

Discussions and conclusions: Generalizability of the findings from the RCT could be limited when the RCT sample does not well represent the target population.

Scientific significance: Application of generalizability weights can be a potentially useful tool to improve generalizability of RCT findings. (Am J Addict 2018;27:231-237).

Publication types

  • Randomized Controlled Trial
  • Research Support, N.I.H., Extramural

MeSH terms

  • Adult
  • Bias
  • Computer Communication Networks
  • Data Collection / methods
  • Education, Distance / methods*
  • Female
  • Generalization, Psychological
  • Help-Seeking Behavior*
  • Humans
  • Male
  • Middle Aged
  • Patient Education as Topic / methods*
  • Patient Selection
  • Substance-Related Disorders* / psychology
  • Substance-Related Disorders* / therapy
  • Treatment Outcome