Categorization is fundamental to our perception and understanding of the environment. However, little is known about the neural bases underlying the categorization of sounds. Using human functional magnetic resonance imaging (fMRI) we compared the brain responses to a category discrimination task with an auditory discrimination task using identical sets of sounds. Our stimuli differed along two dimensions: a speech-nonspeech dimension and a fast-slow temporal dynamics dimension. All stimuli activated regions in the primary and nonprimary auditory cortices in the temporal cortex and in the parietal and frontal cortices for the two tasks. When comparing the activation patterns for the category discrimination task to those for the auditory discrimination task, the results show that a core group of regions beyond the auditory cortices, including inferior and middle frontal gyri, dorsomedial frontal gyrus, and intraparietal sulcus, were preferentially activated for familiar speech categories and for novel nonspeech categories. These regions have been shown to play a role in working memory tasks by a number of studies. Additionally, the categorization of nonspeech sounds activated left middle frontal gyrus and right parietal cortex to a greater extent than did the categorization of speech sounds. Processing the temporal aspects of the stimuli had a greater impact on the left lateralization of the categorization network than did other factors, particularly in the inferior frontal gyrus, suggesting that there is no inherent left hemisphere advantage in the categorical processing of speech stimuli, or for the categorization task itself.