Semantic memory refers to our long-term knowledge of word and object meaning. There is increasing evidence that rather than being a passive warehouse of knowledge, semantic memory is a dynamic system whose effectiveness relies on the coordination of multiple components distributed across a large network of cortical regions. Damage to one or more of these components produces distinct profiles of impairment in aphasia and dementia. Furthermore, such differences are associated with different responses to behavioral treatment. That is, effective treatment for semantically based language disorders in aphasia may have very limited success in dementia. We argue that treatment specificity demands a comprehensive understanding of the structure of semantic memory and the nature of its compromise. Here, we review several neuroanatomically informed theories of semantic organization with respect to the effects of semantic impairment on language processing in aphasia and neurodegenerative disease.