Switching between two tasks results in switch costs, which are increased error rates and response times in comparison to repeating a task. Switch costs are attributed to a change in task set, which is the internalized rule of how to respond to a stimulus. However, it is not clear if this is because the instruction about which task to perform has changed, or because a programmed response has changed. We examined this question by changing the instruction about whether to perform a pro or an antisaccade to a stimulus, before or after the stimulus was presented. As a saccade response is specified by instruction plus stimulus position, changing the instruction after the stimulus was present resulted in a change in the specified response, whereas changing the instruction beforehand did not. Three experiments investigated; (i) if changing instruction alone or changing the specified response produced switch costs; (ii) if predictability of switching instruction influenced switch costs; and (iii) if predictability of stimulus position influenced switch costs. Regardless of instruction or stimulus predictability, switch costs for both pro and antisaccades consistently resulted if the specified response switched. This suggests that a pro or antisaccade motor program was automatically programmed based on a presented instruction and stimulus position. Therefore, the given physical information drove switch costs, even if subjects could predict a change in task. This study demonstrates that switch costs result if changing an instruction changes a programmed response.