Chronic cytomegalovirus (CMV) infection leads to long-term maintenance of extraordinarily large CMV-specific T cell populations. The magnitude of this so-called 'memory inflation' is thought to mainly depend on antigenic stimulation during the chronic phase of infection. However, by mapping the long-term development of CD8+ T cell families derived from single naive precursors, we find that fate decisions made during the acute phase of murine CMV infection can alter the level of memory inflation by more than 1,000-fold. Counterintuitively, a T cell family's capacity for memory inflation is not determined by its initial expansion. Instead, those rare T cell families that dominate the chronic phase of infection show an early transcriptomic signature akin to that of established T central memory cells. Accordingly, a T cell family's long-term dominance is best predicted by its early content of T central memory precursors, which later serve as a stem-cell-like source for memory inflation.