Performance of a receptive language test among young children in Madagascar

Abstract

Language tests developed and validated in one country may lose their desired properties when translated for use in another, possibly resulting in misleading estimates of ability. Using Item Response Theory (IRT) methodology, we assess the performance of a test of receptive vocabulary, the U.S.-validated Peabody Picture Vocabulary Test-Third Edition (PPVT-III), when translated, adapted, and administered to children 3 to 10 years of age in Madagascar (N = 1372), in the local language (Malagasy). Though Malagasy is considered a single language, there are numerous dialects spoken in Madagascar. Our findings were that test scores were positively correlated with age and indicators of socio-economic status. However, over half (57/96) of items evidenced unexpected response variation and/or bias by local dialect spoken. We also encountered measurement error and reduced differentiation among person abilities when we used the publishers' recommended stopping rules, largely because we lost the original item ordering by difficulty when we translated test items into Malagasy. Our results suggest that bias and testing inefficiency introduced from the translation of the PPVT can be significantly reduced with the use of methods based on IRT at both the pre-testing and analysis stages. We explore and discuss implications for cross-cultural comparisons of internationally recognized tests, such as the PPVT.

Fig 1. Standard error of measurement for the Rasch model ability estimates in 2007 and 2011.

In 2007, the standard error on children’s estimated ability increased as the number of item sets administered to the children decreased. In 2011, the standard error on children’s estimated ability followed the expected pattern of increasing error at the extremes of low and high estimated ability.

Fig 2. Item characteristic curves (ICC) in 2011 for items 52, 86 & 89.

The curves represent the probability of a correct response to the indicated item as a function of ability of the respondent. Deviations of the empirical ICC (observed data) from the modeled ICC are indicative of a lack of fit of the item. For example, a flat curve for the observed data is evidence of positive infit (too much variation), and a curve with a steep transition from low to high ability is evidence of negative infit (too little variation). Item 89 (river) is an example of an item with excellent fit (observed data fits the model). Item 52 (huge) is an example with negative infit and item 86 (tropical) is an example with positive infit.

Fig 3. Item characteristic curves for items 32 & 51 with dialect DIF in 2011.

The ICC for item 51 (jogging) is indicative of an item that favored children who spoke official Malagasy. The probability of success is higher for this group (curve labeled “Item 51 Model: Malagasy”) than for the group who spoke a local dialect (curve labeled “Item 51 Model: Dialect”). Item 32 (goat) is an example of an item that favored a local dialect. Item 32 also demonstrated significant positive infit for the children who spoke official Malagasy as shown by the deviation of the empirical ICC (curve labeled “Item 32 Observed: Malagasy”) from the modeled ICC (curve labeled “Item 32 Model: Malagasy”).