It is increasingly argued that cancer stem cells are not a cellular phenotype but rather a transient state that cells can acquire, either through intrinsic signaling cascades or in response to environmental cues. While cancer stem cell plasticity is generally associated with increased aggressiveness and treatment resistance, we set out to thoroughly investigate the impact of different rates of plasticity on early and late tumor growth dynamics and the response to therapy. We develop an agent-based model of cancer stem cell driven tumor growth, in which plasticity is defined as a spontaneous transition between stem and nonstem cancer cell states. Simulations of the model show that plasticity can substantially increase tumor growth rate and invasion. At high rates of plasticity, however, the cells get exhausted and the tumor will undergo spontaneous remission in the long term. In a series of in silico trials, we show that such remission can be facilitated through radiotherapy. The presented study suggests that stem cell plasticity has rather complex, nonintuitive implications on tumor growth and treatment response. Further theoretical, experimental, and integrated studies are needed to fully decipher cancer stem cell plasticity and how it can be harnessed for novel therapeutic approaches.