Empirical performance of a self-controlled cohort method: lessons for developing a risk identification and analysis system

Drug Saf. 2013 Oct;36 Suppl 1:S95-106. doi: 10.1007/s40264-013-0101-3.


Background: Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The self-controlled cohort method, which compares the post-exposure outcome rate with the pre-exposure rate among an exposed cohort, has been proposed as a potential approach for risk identification but its performance has not been fully assessed.

Objectives: To evaluate the performance of the self-controlled cohort method as a tool for risk identification in observational healthcare data.

Research design: The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively.

Measures: Method performance was evaluated through area under ROC curve (AUC), bias, and coverage probability.

Results: The self-controlled cohort design achieved strong predictive accuracy across the outcomes and databases under study, with the top-performing settings exceeding AUC >0.76 in all scenarios. However, the estimates generated were observed to be highly biased with low coverage probability.

Conclusions: If the objective for a risk identification system is one of discrimination, the self-controlled cohort method shows promise as a potential tool for risk identification. However, if a system is intended to generate effect estimates to quantify the magnitude of potential risks, the self-controlled cohort method may not be suitable, and requires substantial calibration to be properly interpreted under nominal properties.

MeSH terms

  • Area Under Curve
  • Bias
  • Cohort Studies*
  • Drug-Related Side Effects and Adverse Reactions / diagnosis*
  • Humans
  • Probability
  • Research Design*
  • Risk Assessment / methods*