Glaucoma is not only an eye disease but is also associated with degeneration of brain structures. We now investigated the pattern of visual and non-visual brain structural changes in 25 primary open angle glaucoma (POAG) patients and 25 age-gender-matched normal controls using T1-weighted imaging. MRI images were subjected to volume-based analysis (VBA) and surface-based analysis (SBA) in the whole brain as well as ROI-based analysis of the lateral geniculate nucleus (LGN), visual cortex (V1/2), amygdala and hippocampus. While VBA showed no significant differences in the gray matter volumes of patients, SBA revealed significantly reduced cortical thickness in the right frontal pole and ROI-based analysis volume shrinkage in LGN bilaterally, right V1 and left amygdala. Structural abnormalities were correlated with clinical parameters in a subset of the patients revealing that the left LGN volume was negatively correlated with bilateral cup-to-disk ratio (CDR), the right LGN volume was positively correlated with the mean deviation of the right visual hemifield, and the right V1 cortical thickness was negatively correlated with the right CDR in glaucoma. These results demonstrate that POAG affects both vision-related structures and non-visual cortical regions. Moreover, alterations of the brain visual structures reflect the clinical severity of glaucoma.