Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score-Based Stratification Approach

Health Serv Res. 2018 Aug;53(4):2567-2590. doi: 10.1111/1475-6773.12752. Epub 2017 Aug 21.


Objective/study question: To estimate and compare sample average treatment effects (SATE) and population average treatment effects (PATE) of a resident duty hour policy change on patient and resident outcomes using data from the Flexibility in Duty Hour Requirements for Surgical Trainees Trial ("FIRST Trial").

Data sources/study setting: Secondary data from the National Surgical Quality Improvement Program and the FIRST Trial (2014-2015).

Study design: The FIRST Trial was a cluster-randomized pragmatic noninferiority trial designed to evaluate the effects of a resident work hour policy change to permit greater flexibility in scheduling on patient and resident outcomes. We estimated hierarchical logistic regression models to estimate the SATE of a policy change on outcomes within an intent-to-treat framework. Propensity score-based poststratification was used to estimate PATE.

Data collection/extraction methods: This study was a secondary analysis of previously collected data.

Principal findings: Although SATE estimates suggested noninferiority of outcomes under flexible duty hour policy versus standard policy, the noninferiority of a policy change was inconclusively noninferior based on PATE estimates due to imprecision.

Conclusions: Propensity score-based poststratification can be valuable tools to address trial generalizability but may yield imprecise estimates of PATE when sparse strata exist.

Keywords: Resident duty hours; generalizability; medical education; propensity score methods; surgical outcomes.

Publication types

  • Pragmatic Clinical Trial
  • Research Support, Non-U.S. Gov't

MeSH terms

  • General Surgery / education
  • Humans
  • Internship and Residency
  • Organizational Innovation*
  • Policy Making*
  • Propensity Score*
  • Workload / standards*