Herpes simplex encephalitis (HSE) is a severe neurological disease that often leads to persistent cognitive deficits in survivors. Memory and naming impairments have been reported most, although direct association between memory and naming performance and disease-related atrophy has not yet been demonstrated in vivo for a larger sample of patients. In the present work, a voxel-based morphometry (VBM) analysis was conducted on 3T magnetic resonance imaging (MRI) of 13 HSE survivors. The gray matter density values were correlated with scores indicating verbal memory decline, as well as errors/omissions in picture naming; both were obtained through neuropsychological assessment. Analysis of individual lesion patterns revealed a considerable inter-individual variability, mainly with atrophy in the basal forebrain, adjacent frontal cortex, medial and lateral temporal cortex, insula and thalamus. The neuropsychological data analysis revealed correlation between verbal memory decline and atrophy especially in the left hippocampal region, whereas naming problems were associated with gray matter loss especially in the lateral temporal lobe, the thalamus and the left insula. These results confirm, for the first time, the assumptions of earlier studies about the considerable variability of individual lesion patterns in HSE in a whole-brain approach in vivo, and thus the anatomical validity of VBM.