Functional magnetic resonance imaging (fMRI) studies have identified category-selective regions in ventral occipito-temporal cortex that respond preferentially to faces and other objects. The extent to which these patterns of activation are modulated by bottom-up or top-down mechanisms is currently unknown. We combined fMRI and dynamic causal modelling to investigate neuronal interactions between occipito-temporal, parietal and frontal regions, during visual perception and visual imagery of faces, houses and chairs. Our results indicate that, during visual perception, category-selective patterns of activation in extrastriate cortex are mediated by content-sensitive forward connections from early visual areas. In contrast, during visual imagery, category-selective activation is mediated by content-sensitive backward connections from prefrontal cortex. Additionally, we report content-unrelated connectivity between parietal cortex and the category-selective regions, during both perception and imagery. Thus, our investigation revealed that neuronal interactions between occipito-temporal, parietal and frontal regions are task- and stimulus-dependent. Sensory representations of faces and objects are mediated by bottom-up mechanisms arising in early visual areas and top-down mechanisms arising in prefrontal cortex, during perception and imagery respectively. Additionally non-selective, top-down processes, originating in superior parietal areas, contribute to the generation of mental images, regardless of their content, and their maintenance in the 'mind's eye'.