Limbic system atrophy and memory dysfunction are common in patients with temporal lobe epilepsy (TLE). However, the relationship between extrahippocampal limbic structures and memory functioning within TLE has received little attention. The present study examined associations of MRI volumetric measurements of the mammillary body, fornix, amygdala, and hippocampus to measures of episodic verbal and visual memory. The Logical Memory and Visual Reproduction subtests from the Wechsler Memory Scale were administered to 47 unilateral TLE patients (25 right, 22 left). Normalized right and left MRI volumes were determined for each patient by cursor tracing 1.5 mm slices from 3D-MRI. Significant associations were found between left hippocampal volume and the immediate, delayed, and percent retention scores of the Logical Memory Test; between the left mammillary body volume and the Logical Memory Test delayed and percent retention scores; immediate Visual Reproduction performance was significantly related to the right and left amygdala volumes, and right mammillary body volume; only the right amygdala and right mammillary body volume were associated with the delayed Visual Reproduction trial. However, neither right nor left hippocampal volumes were related to visual memory performance. Multiple limbic system structural volumes were independently associated with verbal and nonverbal memory performance. Results suggest that visual memory, as measured by the Visual Reproduction Test, may be uniquely associated with extrahippocampal volumes in patients with TLE.