Validation of multilevel regression and poststratification methodology for small area estimation of health indicators from the behavioral risk factor surveillance system

Abstract

Small area estimation is a statistical technique used to produce reliable estimates for smaller geographic areas than those for which the original surveys were designed. Such small area estimates (SAEs) often lack rigorous external validation. In this study, we validated our multilevel regression and poststratification SAEs from 2011 Behavioral Risk Factor Surveillance System data using direct estimates from 2011 Missouri County-Level Study and American Community Survey data at both the state and county levels. Coefficients for correlation between model-based SAEs and Missouri County-Level Study direct estimates for 115 counties in Missouri were all significantly positive (0.28 for obesity and no health-care coverage, 0.40 for current smoking, 0.51 for diabetes, and 0.69 for chronic obstructive pulmonary disease). Coefficients for correlation between model-based SAEs and American Community Survey direct estimates of no health-care coverage were 0.85 at the county level (811 counties) and 0.95 at the state level. Unweighted and weighted model-based SAEs were compared with direct estimates; unweighted models performed better. External validation results suggest that multilevel regression and poststratification model-based SAEs using single-year Behavioral Risk Factor Surveillance System data are valid and could be used to characterize geographic variations in health indictors at local levels (such as counties) when high-quality local survey data are not available.

Keywords: American Community Survey; Behavioral Risk Factor Surveillance System; Missouri County-Level Study; external validation; multilevel regression and poststratification; small area estimation.

Figure 1

Comparison of county-level geographic variations in chronic obstructive pulmonary disease prevalence among adults aged ≥18 years, United States, 2011. A) Unweighted model-based small area estimates from Behavioral Risk Factor Surveillance System data; B) direct survey estimates from the Missouri County-Level Study.

Figure 2

Comparison of geographic variations in county-level prevalence of uninsured adults aged 18–64 years, United States, 2011. A) Unweighted model-based small area estimates from Behavioral Risk Factor Surveillance System data; B) model-based estimates from the Census Bureau’s Small Area Health Insurance Estimates (SAHIE) program.