Inferotemporal (IT) neurons exhibit a substantial degree of invariance with respect to translation of images used as visual stimuli. Through theoretical and computer-modeling methods, we show how translation-invariant receptive fields, like those of IT neurons, can be generated from the responses of V4 neurons if the effects of attention are taken into account. The model incorporates a recently reported form of attention-dependent gain modulation in V4 and produces IT receptive fields that shift so they are centered at the point where attention is directed. Receptive fields of variable, attention-controlled spatial scale are obtained when the mechanism is extended to scale-dependent attentional gain fields. The results indicate that gain modulation may play analogous roles in the dorsal and ventral visual pathways, generating transformations from retinal coordinates to body- and object-centered systems, respectively.