Examining multiple sources of differential item functioning on the Clinician & Group CAHPS® survey

Abstract

Objective: To evaluate psychometric properties of a widely used patient experience survey.

Data sources: English-language responses to the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) survey (n = 12,244) from a 2008 quality improvement initiative involving eight southern California medical groups.

Methods: We used an iterative hybrid ordinal logistic regression/item response theory differential item functioning (DIF) algorithm to identify items with DIF related to patient sociodemographic characteristics, duration of the physician-patient relationship, number of physician visits, and self-rated physical and mental health. We accounted for all sources of DIF and determined its cumulative impact.

Principal findings: The upper end of the CG-CAHPS® performance range is measured with low precision. With sensitive settings, some items were found to have DIF. However, overall DIF impact was negligible, as 0.14 percent of participants had salient DIF impact. Latinos who spoke predominantly English at home had the highest prevalence of salient DIF impact at 0.26 percent.

Conclusions: The CG-CAHPS® functions similarly across commercially insured respondents from diverse backgrounds. Consequently, previously documented racial and ethnic group differences likely reflect true differences rather than measurement bias. The impact of low precision at the upper end of the scale should be clarified.

Figure 1

Test Characteristic (A) and Information (B) Curves. Notes. (a) The test characteristic curve, which is a plot of the most likely standard score associated with each level of overall primary care experience. This curve shows that the distribution of items is not uniform across the range measured by the scale, as the slope of the curve is higher to the left of about 0 than to the right. This finding suggests problems with using standard scores in regression models; item response theory (IRT) scores should be used instead (Crane et al. 2008a). (b) The test information curve (black curve) and the standard error of measurement curve (gray curve) at each level of overall primary care experience are shown. These curves further document the uneven distribution of items across the scale. Test information is adequate to the left of 0 but drops to the right of 0. This is reflected by standard error of measurement curve, which is characterized by large amounts of measurement error at the top end of the scale. This figure is analogous to the alpha coefficient commonly reported using classical test theory. Unlike classical test theory, however, IRT does not assume that measurement precision is consistent across the entire scale, and it does not summarize measurement with a single omnibus statistic such as the alpha coefficient. See McDonald (1999) for further discussion.

Figure 2

Differential Item Functioning (DIF) Impact, by Patient Race, Ethnicity, and Primary Language Spoken at Home. Notes. This figure plots the distributions of differences between naive scores ignoring DIF and scores that account for all sources of DIF across the six race-ethnicity groups evaluated in the study. Differences of 0 indicate no DIF impact. We use the median standard error of measurement for the scale to demarcate levels of DIF impact that can be distinguished from negligible effects (dark vertical lines); DIF greater than this level is referred to as “salient” DIF. The box represents the 25th and 75th percentiles of the distribution, and the whiskers represent 1½ times the distribution of the box. Observations more extreme than the whiskers are shown with dots. This graph depicts that the box denoting the interquartile range is very close to 0, and that of the whiskers are well within the dark vertical lines denoting the standard error of measurement. A small and negligible number of people have salient DIF impact when accounting for all the sources of DIF considered here. See text for further details.