Background and objectives: There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population.
Methods: The sample of the web-based SUD intervention (Therapeutic Education System vs. Treatment-as-usual; n = 507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, we computed weighted treatment effects on retention and abstinence.
Results: Substantial differences between the RCT sample and the target population was demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p < .001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight.
Discussions and conclusions: Generalizability of the findings from the RCT could be limited when the RCT sample does not well represent the target population.
Scientific significance: Application of generalizability weights can be a potentially useful tool to improve generalizability of RCT findings. (Am J Addict 2018;27:231-237).
© 2018 American Academy of Addiction Psychiatry.