A new adaptive testing algorithm for shortening health literacy assessments

Abstract

Background: Low health literacy has a detrimental effect on health outcomes, as well as ability to use online health resources. Good health literacy assessment tools must be brief to be adopted in practice; test development from the perspective of item-response theory requires pretesting on large participant populations. Our objective was to develop a novel classification method for developing brief assessment instruments that does not require pretesting on large numbers of research participants, and that would be suitable for computerized adaptive testing.

Methods: We present a new algorithm that uses principles of measurement decision theory (MDT) and Shannon's information theory. As a demonstration, we applied it to a secondary analysis of data sets from two assessment tests: a study that measured patients' familiarity with health terms (52 participants, 60 items) and a study that assessed health numeracy (165 participants, 8 items).

Results: In the familiarity data set, the method correctly classified 88.5% of the subjects, and the average length of test was reduced by about 50%. In the numeracy data set, for a two-class classification scheme, 96.9% of the subjects were correctly classified with a more modest reduction in test length of 35.7%; a three-class scheme correctly classified 93.8% with a 17.7% reduction in test length.

Conclusions: MDT-based approaches are a promising alternative to approaches based on item-response theory, and are well-suited for computerized adaptive testing in the health domain.

Figure 1

Example of questions from study 1 that tested consumers' familiarity with terms ("fascia") and concepts ("cancer").

Figure 2

Entropy threshold vs. Average question count and Error rate using start criterion SA (dotted lines) and SB(solid lines).

Figure 3

Entropy threshold vs. Average question count and error count, for entropy thresholds in range 0 to 0.01 with start criterion SA.

Figure 4

Entropy threshold vs. Average question count and error count using start criterion SA, for data set 2 with two-class classification scheme.

Figure 5

Entropy threshold vs. Average question count and error count using start criterion SA, for data set 2 with three-class classification scheme.

Figure 6

Sensitivity and specificity of the algorithm for Data set 2 with two class classification scheme. The top ROC curve shows the sensitivity and false positive rate on the numeracy data set at an entropy threshold of 0 (area under the curve = 0.96); the lower curve shows very little decrement in performance at an entropy of 0.6 (area under the curve = 0.93).