The conventional view of bacterial adaptation emphasizes the importance of rapidly evolved changes that are highly repeatable in response to similar environments and subject to loss in the absence of selection. Consequently, genetic variation is not expected to persist over long time scales for these organisms. Here, we show that a geographically widespread gene content polymorphism has surprisingly been maintained for tens of millions of years of diversification of the multicellular cyanobacterium Fischerella thermalis. The polymorphism affects gas permeability of the heterocyst-the oxygen-sensitive, nitrogen-fixing cell produced by these bacteria-and spatial variation in temperature favours alternative alleles due to thermodynamic effects on both heterocyst function and organism fitness at physiological temperature extremes. Whether or not ancient balancing selection plays a generally important role in the maintenance of microbial diversity remains to be investigated.