Avoiding randomization failure in program evaluation, with application to the Medicare Health Support program

Popul Health Manag. 2011 Feb:14 Suppl 1:S11-22. doi: 10.1089/pop.2010.0074.

Abstract

We highlight common problems in the application of random treatment assignment in large-scale program evaluation. Random assignment is the defining feature of modern experimental design, yet errors in design, implementation, and analysis often result in real-world applications not benefiting from its advantages. The errors discussed here cover the control of variability, levels of randomization, size of treatment arms, and power to detect causal effects, as well as the many problems that commonly lead to post-treatment bias. We illustrate these issues by identifying numerous serious errors in the Medicare Health Support evaluation and offering recommendations to improve the design and analysis of this and other large-scale randomized experiments.

Publication types

  • Evaluation Study
  • Research Support, Non-U.S. Gov't
  • Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

  • Government Programs / economics*
  • Government Programs / statistics & numerical data
  • Humans
  • Medicare / economics*
  • Medicare / statistics & numerical data
  • Pilot Projects
  • Program Evaluation / economics*
  • Program Evaluation / statistics & numerical data
  • Randomized Controlled Trials as Topic / economics*
  • Randomized Controlled Trials as Topic / statistics & numerical data
  • Research Design*
  • Selection Bias*
  • United States