Residual tests in the analysis of planned contrasts: Problems and solutions

Abstract

It is current practice that researchers testing specific, theory-driven predictions do not only use a planned contrast to model and test their hypotheses, but also test the residual variance (the C+R approach). This analysis strategy relies on work by Abelson and Prentice (1997), who suggested that the result of a planned contrast needs to be interpreted in light of the variance that is left after the variance explained by the contrast has been subtracted from the variance explained by the factors of the statistical model. Unfortunately, the C + R approach leads to 6 fundamental problems. In particular, the C + R approach (a) relies on the interpretation of a nonsignificant result as evidence for no effect, (b) neglects the impact of sample size, (c) creates problems for a priori power analyses, (d) may lead to significant effects that lack a meaningful interpretation, (e) may give rise to misinterpretations, and (f) is inconsistent with the interpretation of other statistical analyses. Given these flaws, researchers should refrain from testing the residual variance when conducting planned contrasts. Single contrasts, Bayes factors, and likelihood ratios provide reasonable alternatives that are less problematic.