Apolipoprotein E ε4 allele (ApoE4) has been associated with increased risk of sporadic Alzheimer's disease (AD) and of conversion from mild cognitive impairment to AD. But the underlying mechanism of ApoE4 affecting brain atrophy and cognition is not fully understood. We investigated the effect of ApoE4 on amyloid beta (Aβ) protein burden and its correlation with the structure change of hippocampus and cortex, cognitive and behavioral changes in ApoE4 transgenic mice. Male ApoE4 transgenic mice and age-matched control mice at age 12 months and 24 months were tested in the Morris Water Maze (MWM). Brain volume changes (including whole brain, hippocampus, cortex, total ventricles and caudate putamen) were assessed by using small animal 7T-MRI. Aβ level was assessed by immunohistochemistry (IHC) and immunoprecipitation/western blot. In MWM, escape latency was longer and time spent in the target quadrant was shorter in aged ApoE4 mice (12- and 24-month-old), suggesting age- and ApoE4-dependent visuospatial deficits. Atrophy on MRI was prominent in the hippocampus (p=0.039) and cortex (p=0.013) of ApoE4 mice (24-month-old) as compared to age-matched control mice. IHC revealed elevated Aβ deposition in the hippocampus. Consistently, both soluble and insoluble Aβ aggregates were increased in aged ApoE4 mice. This increase was correlated inversely with hippocampal atrophy and cognitive deficits. These data give further evidence that ApoE4 plays an important role in brain atrophy and memory impairment by modulating amyloid production and deposition.