Stem cells are remarkably small. Whether small size is important for stem cell function is unknown. We find that hematopoietic stem cells (HSCs) enlarge under conditions known to decrease stem cell function. This decreased fitness of large HSCs is due to reduced proliferation and was accompanied by altered metabolism. Preventing HSC enlargement or reducing large HSCs in size averts the loss of stem cell potential under conditions causing stem cell exhaustion. Last, we show that murine and human HSCs enlarge during aging. Preventing this age-dependent enlargement improves HSC function. We conclude that small cell size is important for stem cell function in vivo and propose that stem cell enlargement contributes to their functional decline during aging.