Epigenetic information undergoes extensive reprogramming in the germline between generations. This reprogramming may be essential to establish a developmental ground state in the zygote. We show that mutants in spr-5, the Caenorhabditis elegans ortholog of the H3K4me2 demethylase LSD1/KDM1, exhibit progressive sterility over many generations. This sterility correlates with the misregulation of spermatogenesis-expressed genes and transgenerational accumulation of the histone modification dimethylation of histone H3 on lysine 4 (H3K4me2). This suggests that H3K4me2 can serve as a stable epigenetic memory, and that erasure of H3K4me2 by LSD/KDM1 in the germline prevents the inappropriate transmission of this epigenetic memory from one generation to the next. Thus, our results provide direct mechanistic insights into the processes that are required for epigenetic reprogramming between generations.