The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions

Eval Rev. 2016 Jun;40(3):167-198. doi: 10.1177/0193841X16663414. Epub 2016 Aug 23.

Abstract

Objective: In this article, we examine whether a well-executed comparative interrupted time series (CITS) design can produce valid inferences about the effectiveness of a school-level intervention. This article also explores the trade-off between bias reduction and precision loss across different methods of selecting comparison groups for the CITS design and assesses whether choosing matched comparison schools based only on preintervention test scores is sufficient to produce internally valid impact estimates.

Research design: We conduct a validation study of the CITS design based on the federal Reading First program as implemented in one state using results from a regression discontinuity design as a causal benchmark.

Results: Our results contribute to the growing base of evidence regarding the validity of nonexperimental designs. We demonstrate that the CITS design can, in our example, produce internally valid estimates of program impacts when multiple years of preintervention outcome data (test scores in the present case) are available and when a set of reasonable criteria are used to select comparison organizations (schools in the present case).

Keywords: comparative interrupted time series analysis; design replication; quasi-experimental methods; validation study; within-study comparison.

Publication types

  • Comparative Study
  • Review
  • Validation Study

MeSH terms

  • Female
  • Humans
  • Interrupted Time Series Analysis*
  • Male
  • Research Design*
  • Schools / organization & administration*
  • United States