Traditional validation processes for psychological surveys tend to focus on analyzing item responses instead of the cognitive processes that participants use to generate these responses. When screening for invalid responses, researchers typically focus on participants who manipulate their answers for personal gain or respond carelessly. In this paper, we introduce a new invalid response process, discordant responding, that arises when participants disagree with the use of the survey and discuss similarities and differences between this response style and protective responding. Results show that nearly all participants reflect on the intended uses of an assessment when responding to items and may decline to respond or modify their responses if they are not comfortable with the way the results will be used. Incidentally, we also find that participants may misread survey instructions if they are not interactive. We introduce a short screener to detect invalid responses, the discordant response identifiers (DRI), which provides researchers with a simple validity tool to use when validating surveys. Finally, we provide recommendations about how researchers may use these findings to design surveys that reduce this response manipulation in the first place.
Keywords: assessment; attention; honesty; participant motivation; survey; survey use; validity.