Recent results indicate that crossmodal interactions can affect activity in cortical regions traditionally regarded as "unimodal." Previously we found that combining touch on one hand with visual stimulation in the anatomically corresponding hemifield could boost responses in contralateral visual cortex. Here we manipulated which visual hemifield corresponded to the location of the stimulated hand, by changing gaze direction such that right-hand touch could now arise in either the left or right visual field. Crossmodal effects on visual cortex switched from one hemisphere to the other, depending on gaze direction, regardless of whether the hand was seen. This indicates that crossmodal influences of touch upon visual cortex depend on spatial alignment for the multimodal stimuli, with gaze posture taken into account.