Evidence on whether genetic predictors of Alzheimer disease (AD) also predict memory decline is inconsistent, and limited data are available for African ancestry populations. For 8253 non-Hispanic white (NHW) and non-Hispanic black (NHB) Health and Retirement Study participants with memory scores measured 1 to 8 times between 1998 and 2012 (average baseline age=62), we calculated weighted polygenic risk scores [AD Genetic Risk Score (AD-GRS)] using the top 22 AD-associated loci, and an alternative score excluding apolipoprotein E (APOE) (AD-GRSexAPOE). We used generalized linear models with AD-GRS-by-age and AD-GRS-by-age interactions (age centered at 70) to predict memory decline. Average NHB decline was 26% faster than NHW decline (P<0.001). Among NHW, 10% higher AD-GRS predicted faster memory decline (linear β=-0.058 unit decrease over 10 y; 95% confidence interval,-0.074 to -0.043). AD-GRSexAPOE also predicted faster decline for NHW, although less strongly. Among NHB, AD-GRS predicted faster memory decline (linear β=-0.050; 95% confidence interval, -0.106 to 0.006), but AD-GRSexAPOE did not. Our nonsignificant estimate among NHB may reflect insufficient statistical power or a misspecified AD-GRS among NHB as an overwhelming majority of genome-wide association studies are conducted in NHW. A polygenic score based on previously identified AD loci predicts memory loss in US blacks and whites.