Standard error in the Jacobson and Truax Reliable Change Index: the "classical approach" leads to poor estimates

Abstract

Different authors have used different estimates of variability in the denominator of the Reliable Change Index (RCI). Maassen attempts to clarify some of the differences and the assumptions underlying them. In particular he compares the 'classical' approach using an estimate S(Ed) supposedly based on measurement error alone with an estimate S(Diff) based on the variability of observed differences in a population that should have no true change. Maassen concludes that not only is S(Ed) based on classical theory, but it properly estimates variability due to measurement error and practice effect while S(Diff) overestimates variability by accounting twice for the variability due to practice. Simulations show Maassen to be wrong on both accounts. With an error rate nominally set to 10%, RCI estimates using S(Diff) wrongly declare change in 10.4% and 9.4% of simulated cases without true change while estimates using S(Ed) wrongly declare change in 17.5% and 12.3% of the simulated cases (p < .000000001 and p < .008, respectively). In the simulation that separates measurement error and practice effects, SEd estimates the variability of change due to measurement error to be .34, when the true variability due to measurement error was .014. Neuropsychologists should not use SEd in the denominator of the RCI.