Comorbidity with anxiety disorder is a relatively common occurrence in major depressive disorder. However, the unique and shared neuroanatomical characteristics of depression and anxiety disorders have not been fully identified. The aim of this study was to identify gray matter abnormalities and their clinical correlates in depressive patients with and without anxiety disorders. We applied voxel-based morphometry and region-of-interest analyses of gray matter volume (GMV) in normal controls (NC group, n = 28), depressive patients without anxiety disorder (DP group, n = 18), and depressive patients with anxiety disorder (DPA group, n = 20). The correlations between regional GMV and clinical data were analyzed. The DP group showed decreased GMV in the left insula (INS) and left triangular part of the inferior frontal gyrus when compared to the NC group. The DPA group showed greater GMV in the midbrain, medial prefrontal cortex, and primary motor/somatosensory cortex when compared to the NC group. Moreover, the DPA group showed greater GMV than the DP group in the frontal, INS, and temporal lobes. Most gray matter anomalies were significantly correlated with depression severity or anxiety symptoms. These correlations were categorized into 4 trend models, of which 3 trend models (ie, Models I, II, and IV) revealed the direction of the correlation between regional GMV and depression severity to be the opposite of that between regional GMV and anxiety symptoms. Importantly, the left INS showed a trend Model I, which might be critically important for distinguishing depressive patients with and without anxiety disorder. Our findings of gray matter abnormalities, their correlations with clinical data, and the trend models showing opposite direction may reflect disorder-specific symptom characteristics and help explain the neurobiological differences between depression and anxiety disorder.