Task switching studies revealed that the usual response-repetition benefit is abolished and often reversed if the task switches. According to episodic binding accounts, performing responses strengthens task-specific bindings, leading to response-repetition benefits in task repetitions, whereas such bindings can lead to interference (i.e., costs of "unbinding") in task switches. An alternative account assumes that responses are generally inhibited after execution but that the assumed sequential carryover of response inhibition is overcompensated by positive priming of stimulus category in task repetitions (resulting in a positive net effect in response-repetition conditions). In the present study, we manipulated task-cue modality (visual vs. auditory) to introduce a variation of encoding and retrieval context, which should vary the strength of episodic bindings. Across two experiments (Experiment 1A, showing the initial evidence, and Experiment 1B, providing a successful replication), we found that the response-repetition benefit in task repetitions was substantially larger with repeated cue modality than with changed cue modality, suggesting that cue modality primes retrieval of task-specific stimulus categories and responses. However, the observed response-repetition cost in task switches remained unaffected by this contextual change. This data pattern suggests a hybrid account, assuming that response-repetition benefits are driven by episodic bindings, whereas response-repetition costs are primarily due to (non-episodic) carryover of response inhibition.