Our cross-sectional study showed that the interaction between apolipoprotein E4 (ApoE4) and angiotensin converting enzyme (ACE) inhibitors was associated with Alzheimer's disease (AD). The aim of this longitudinal study was to differentiate whether ACE inhibitors accelerate or reduce the risk of AD in the context of ApoE alleles. Using the longitudinal data from the National Alzheimer's Coordinating Center (NACC) with ApoE genotyping and documentation of ACE inhibitors use, we found that in the absence of ApoE4, subjects who had been taking central ACE inhibitor use (χ2 test: 21% versus 27%, p = 0.0002) or peripheral ACE inhibitor use (χ2 test: 13% versus 27%, p < 0.0001) had lower incidence of AD compared with those who had not been taking an ACE inhibitor. In contrast, in the presence of ApoE4, there was no such association between ACE inhibitor use and the risk of AD. After adjusting for the confounders, central ACE inhibitor use (OR = 0.68, 95% CI = 0.55, 0.83, p = 0.0002) or peripheral ACE inhibitor use (OR = 0.33, 95% CI = 0.33, 0.68, p < 0.0001) still remained inversely associated with a risk of developing AD in ApoE4 non-carriers. In conclusion, ACE inhibitors, especially peripherally acting ones, were associated with a reduced risk of AD in the absence of ApoE4, but had no such effect in those carrying the ApoE4 allele. A double-blind clinical trial should be considered to determine the effect of ACE inhibitors on prevention of AD in the context of ApoE genotype.