Quantitative approaches for the evaluation of implementation research studies

Abstract

Implementation research necessitates a shift from clinical trial methods in both the conduct of the study and in the way that it is evaluated given the focus on the impact of implementation strategies. That is, the methods or techniques to support the adoption and delivery of a clinical or preventive intervention, program, or policy. As strategies target one or more levels within the service delivery system, evaluating their impact needs to follow suit. This article discusses the methods and practices involved in quantitative evaluations of implementation research studies. We focus on evaluation methods that characterize and quantify the overall impacts of an implementation strategy on various outcomes. This article discusses available measurement methods for common quantitative implementation outcomes involved in such an evaluation-adoption, fidelity, implementation cost, reach, and sustainment-and the sources of such data for these metrics using established taxonomies and frameworks. Last, we present an example of a quantitative evaluation from an ongoing randomized rollout implementation trial of the Collaborative Care Model for depression management in a large primary healthcare system.

Keywords: Implementation measurement; Implementation research; Summative evaluation.

Figure 1.

Emphasis and Outcomes Evaluated in Clinical Effectiveness versus Implementation Research Note. Adapted from a slide developed by C. Hendricks Brown.

Figure 2.

Design and Timeline of Randomized Rollout Implementation Trial of CCM Note. CCM = Collaborative Care Model. Clinics will have a staggered start every 3–4 months randomized using a matching scheme. Pre-implementation assessment period is 4 months. Evaluation of CCM implementation will be a minimum of 24 months at each clinic.

Figure 3.

Summative Evaluation Metrics of CCM Implementation Using the RE-AIM Framework Note. CCM = Collaborative Care Model. EHR = electronic health record.