Cognitive neuroscientific research proposes complementary hemispheric asymmetries in naming and recognising visual objects, with a left temporal lobe advantage for object naming and a right temporal lobe advantage for object recognition. Specifically, it has been proposed that the left inferior temporal lobe plays a mediational role linking conceptual information with word forms and vice versa, while the right inferior temporal lobe supports the retrieval of conceptual knowledge from visual input. To test these hypotheses, we administered four behavioural tasks to fifteen patients with temporal lobe brain damage, and correlated their behavioural scores with voxel-based measures of neuronal integrity (signal intensities) in whole-brain analyses. The behavioural paradigms included four tasks. Two were verbal tasks: (a) picture naming requiring the mapping of conceptual knowledge to word forms, (b) semantic categorisation of words requiring the reverse mapping of word forms to conceptual knowledge, and two were visual object tasks with no verbal component, both of which required the retrieval of conceptual information from visual objects, i.e., (c) visual object categorisation and (d) normal and chimera object decisions. Performance on the verbal tasks correlated with the neural integrity of partially overlapping left inferior and anterior temporal lobe regions, while performance on the object tasks correlated with the neural integrity of overlapping regions in right inferior and anterior temporal lobe. These findings support the notion of complementary hemispheric advantages for object naming and recognition, and further suggest that the classical language model emphasising posterior regions in the mapping between word forms and conceptual knowledge should be extended to include left inferior temporal lobe.