A Methodology For Studying Organizational Performance: A Multistate Survey of Front-line Providers

Abstract

Background: Rigorous measurement of organizational performance requires large, unbiased samples to allow inferences to the population. Studies of organizations, including hospitals, often rely on voluntary surveys subject to nonresponse bias. For example, hospital administrators with concerns about performance are more likely to opt-out of surveys about organizational quality and safety, which is problematic for generating inferences.

Objective: The objective of this study was to describe a novel approach to obtaining a representative sample of organizations using individuals nested within organizations, and demonstrate how resurveying nonrespondents can allay concerns about bias from low response rates at the individual-level.

Methods: We review and analyze common ways of surveying hospitals. We describe the approach and results of a double-sampling technique of surveying nurses as informants about hospital quality and performance. Finally, we provide recommendations for sampling and survey methods to increase response rates and evaluate whether and to what extent bias exists.

Results: The survey of nurses yielded data on over 95% of hospitals in the sampling frame. Although the nurse response rate was 26%, comparisons of nurses' responses in the main survey and those of resurveyed nonrespondents, which yielded nearly a 90% response rate, revealed no statistically significant differences at the nurse-level, suggesting no evidence of nonresponse bias.

Conclusions: Surveying organizations via random sampling of front-line providers can avoid the self-selection issues caused by directly sampling organizations. Response rates are commonly misinterpreted as a measure of representativeness; however, findings from the double-sampling approach show how low response rates merely increase the potential for nonresponse bias but do not confirm it.

FIGURE 1.

Distribution of a number of patients per nurse among main survey respondents and nonrespondents. This figure includes nurse respondents employed in hospitals as direct care staff nurses caring for ≤ 15 patients on their last shift.

FIGURE 2.

Recommendations for sampling and survey methods to increase representativeness and reduce nonresponse bias.