Primates emerged about 60 million years ago. Since that time various primate-targeting retroviruses have integrated in the germ line of primate species, and some drifted to fixation. After germ line fixation, continued activity of proviruses resulted in intragenomic spread of so-called endogenous retroviruses (ERVs). Variant ERVs emerged, amplified in the genome and profoundly altered genome structures and potentially functionality. Importantly, ERVs are genome modifiers of exogenous origin. The human genome contains about 8% of sequences of retroviral origin. The human ERVs (HERVs) comprise many distinct families that amplified to copy numbers of up to several thousand. We review here the evolution of several well-characterized HERV families in the human lineage since initial germ line fixation. It is apparent that endogenous retroviruses profoundly affected the genomes of species in the evolutionary lineage leading to Homo sapiens.