Host-parasite associations are shaped by coevolutionary dynamics. One example is the complex fungus-growing ant-microbe symbiosis, which includes ancient host-parasite coevolution. Fungus-growing ants and the fungi they cultivate for food have an antagonistic symbiosis with Escovopsis, a specialized microfungus that infects the ants' fungus gardens. The evolutionary histories of the ant, cultivar and Escovopsis are highly congruent at the deepest phylogenetic levels, with specific parasite lineages exclusively associating with corresponding groups of ants and cultivar. Here, we examine host-parasite specificity at finer phylogenetic levels, within the most derived clade of fungus-growing ants, the leaf-cutters (Atta spp. and Acromyrmex spp.). Our molecular phylogeny of Escovopsis isolates from the leaf-cutter ant-microbe symbiosis confirms specificity at the broad phylogenetic level, but reveals frequent host-switching events between species and genera of leaf-cutter ants. Escovopsis strains isolated from Acromyrmex and Atta gardens occur together in the same clades, and very closely related strains can even infect the gardens of both ant genera. Experimental evidence supports low host-parasite specificity, with phylogenetically diverse strains of Escovopsis being capable of overgrowing all leaf-cutter cultivars examined. Thus, our findings indicate that this host-pathogen association is shaped by the farming ants having to protect their cultivated fungus from phylogenetically diverse Escovopsis garden pathogens.