Musical timbre is a multidimensional property of sound that allows one to distinguish musical instruments. In this paper, studies that explore the cerebral substrate underlying the processing of musical timbre are discussed. Perceptual asymmetries measured in normal participants, deficits of musical timbre perception obtained in brain-damaged patients, as well as results obtained with various neuroimaging methods are reviewed. The findings obtained in all of these studies generally support the predominant involvement of right temporal lobe areas, and more specifically of its anterior part, in processing spectral and temporal envelopes of musical timbre. However, controversies still exist about the contribution of the left temporal lobe in timbre perception. The necessity of comparing data obtained with different perceptual paradigms (same-different discrimination and similarity judgment) and various types of stimuli (single tones and melodies) was emphasized by reporting lesion studies carried out in patients with unilateral temporal lobe lesions. The few neuroimaging studies published in this domain provided additional and complementary findings. Unlike lesion studies that allow us to infer the cerebral structures that are essential for timbre perception, the latter investigations implicate a more distributed neural network in timbre processing that extends along the superior temporal gyrus to include not only anterior but also posterior temporal regions and possibly frontal areas as well.