A long-lived reservoir of latently infected T cells prevents antiretroviral therapy from eliminating HIV-1 infection. Furthering our understanding of the dynamics of latency generation and maintenance is therefore vital to improve treatment outcome. Using mathematical models and experiments, we suggest that the death of latently infected cells brought about by pyroptosis, or to a lesser extent by superinfection, might be key mechanisms to account for the size and composition of the latent reservoir. Pyroptosis is a form of cell death that occurs in a resting (and thus latently infected) T cell when a productively infected cell attempts cell-to-cell transmission of virus. Superinfection of latently infected cells by productive virus could similarly remove those cells through active virus replication and resulting cytopathicity. The mathematical models presented can explain a number of previously published clinical observations including latent reservoir size and the relationships to viral load in acute HIV infection, measurements of the latent reservoir in chronic infection, and the replacement of wild-type virus by CTL escape mutants within the latent reservoir. Basic virus dynamics models of latency that do not take into account pyroptosis, superinfection, or other potential complexities cannot account for the data.