The partitioning and subsequent inheritance of cellular factors like proteins and RNAs is a ubiquitous feature of cell division. However, direct quantitative measures of how such nongenetic inheritance affects subsequent changes in gene expression have been lacking. We tracked families of the yeast Saccharomyces cerevisiae as they switch between two semi-stable epigenetic states. We found that long after two cells have divided, they continued to switch in a synchronized manner, whereas individual cells have exponentially distributed switching times. By comparing these results to a Poisson process, we show that the time evolution of an epigenetic state depends initially on inherited factors, with stochastic processes requiring several generations to decorrelate closely related cells. Finally, a simple stochastic model demonstrates that a single fluctuating regulatory protein that is synthesized in large bursts can explain the bulk of our results.