The brain regions that are critically associated with visual neglect have become intensely disputed. In particular, one study of middle cerebral artery (MCA) stroke patients has claimed that the key brain region associated with neglect is the mid portion of the superior temporal gyrus (STG), on the lateral surface of the right hemisphere, rather than the posterior parietal lobe. Such a result has wide-ranging implications for both our understanding of the normal function these cortical areas and the potential mechanisms underlying neglect. Here, we use novel high resolution MRI protocols to map the lesions of 35 right-hemisphere patients who had suffered either MCA or posterior cerebral artery (PCA) territory stroke. For patients with MCA territory strokes, the critical area involved in all neglect patients was the angular gyrus of the inferior parietal lobe (IPL). Although the STG was damaged in half of our MCA neglect patients, it was spared in the rest. For PCA territory strokes, all patients with neglect had lesions involving the parahippocampal region, on the medial surface of the temporal lobe. PCA patients without neglect did not have damage to this area. We conclude that damage to two posterior regions, one in the IPL and the other in the medial temporal lobe, is associated with neglect. Although some neglect patients do have damage to the STG, our findings challenge the recent influential proposal that lesions of this area are critically associated with neglect. Instead, our results implicate the angular gyrus and parahippocampal region in this role.