Squeezing the balloon: propensity scores and unmeasured covariate balance

Abstract

Objective: To assess the covariate balancing properties of propensity score-based algorithms in which covariates affecting treatment choice are both measured and unmeasured.

Data sources/study setting: A simulation model of treatment choice and outcome.

Study design: Simulation.

Data collection/extraction methods: Eight simulation scenarios varied with the values placed on measured and unmeasured covariates and the strength of the relationships between the measured and unmeasured covariates. The balance of both measured and unmeasured covariates was compared across patients either grouped or reweighted by propensity scores methods.

Principal findings: Propensity score algorithms require unmeasured covariate variation that is unrelated to measured covariates, and they exacerbate the imbalance in this variation between treated and untreated patients relative to the full unweighted sample.

Conclusions: The balance of measured covariates between treated and untreated patients has opposite implications for unmeasured covariates in randomized and observational studies. Measured covariate balance between treated and untreated patients in randomized studies reinforces the notion that all covariates are balanced. In contrast, forced balance of measured covariates using propensity score methods in observational studies exacerbates the imbalance in the independent portion of the variation in the unmeasured covariates, which can be likened to squeezing a balloon. If the unmeasured covariates affecting treatment choice are confounders, propensity score methods can exacerbate the bias in treatment effect estimates.

Keywords: Propensity scores; assumptions; binning; covariate balance; matching; simulation.

Figure 1

Directed Acyclic Graph of Simulation Model Relationships

Figure 2

Percent Reduction (Increase) in Sample Mean Difference (Treated–Untreated) from PS Binning for X_M and R₁, by Unmeasured Covariate Effect Size (α_U1)

Figure 3

Percent Reduction (Increase) in Sample Mean Difference (Treated–Untreated) from PS Binning for X_M and R₁, by Unmeasured Covariate Effect Size (α_U1)

Figure 4

Percent Reduction (Increase) in Sample Mean Difference (Treated–Untreated) from PS Binning for X_M and R₁, by Correlation between X_M, X_U1 (ρ_M,U1) Note. [1] Empty cells in Bins 1 and 5; [2] Empty cells in Bins 1, 2, 4, and 5.

Figure 5

Percent Reduction (Increase) in Sample Mean Difference (Treated–Untreated) from PS Matching for X_M and R₁, by Match Tolerance Factor (±0.1, ±0.01, ±0.001)