Recommendations for evaluation of health care improvement initiatives

Abstract

Intensive efforts are underway across the world to improve the quality of health care. It is important to use evaluation methods to identify improvement efforts that work well before they are replicated across a broad range of contexts. Evaluation methods need to provide an understanding of why an improvement initiative has or has not worked and how it can be improved in the future. However, improvement initiatives are complex, and evaluation is not always well aligned with the intent and maturity of the intervention, thus limiting the applicability of the results. We describe how initiatives can be grouped into 1 of 3 improvement phases-innovation, testing, and scale-up and spread-depending on the degree of belief in the associated interventions. We describe how many evaluation approaches often lead to a finding of no effect, consistent with what has been termed Rossi's Iron Law of Evaluation. Alternatively, we recommend that the guiding question of evaluation in health care improvement be, "How and in what contexts does a new model work or can be amended to work?" To answer this, we argue for the adoption of formative, theory-driven evaluation. Specifically, evaluations start by identifying a program theory that comprises execution and content theories. These theories should be revised as the initiative develops by applying a rapid-cycle evaluation approach, in which evaluation findings are fed back to the initiative leaders on a regular basis. We describe such evaluation strategies, accounting for the phase of improvement as well as the context and setting in which the improvement concept is being deployed. Finally, we challenge the improvement and evaluation communities to come together to refine the specific methods required so as to avoid the trap of Rossi's Iron Law.

Keywords: evaluation; methods; quality improvement.