*The colonization of land by plants fundamentally altered environmental conditions on earth. Plant-mycorrhizal fungus symbiosis likely played a key role in this process by assisting plants to absorb water and nutrients from soil. *Here, in a diverse set of land plants, we investigated the evolutionary histories and functional conservation of three genes required for mycorrhiza formation in legumes and rice (Oryza sativa), DMI1, DMI3 and IPD3. *The genes were isolated from nearly all major plant lineages. Phylogenetic analyses showed that they had been vertically inherited since the origin of land plants. Further, cross-species mutant rescue experiments demonstrated that DMI3 genes from liverworts and hornworts could rescue Medicago truncatula dmi3 mutants for mycorrhiza formation. Yeast two-hybrid assays also showed that bryophyte DMI3 proteins could bind to downstream-acting M. trunculata IPD3 protein. Finally, molecular evolutionary analyses revealed that these genes were under purifying selection for maintenance of their ancestral functions in all mycorrhizal plant lineages. *These results indicate that the mycorrhizal genes were present in the common ancestor of land plants, and that their functions were largely conserved during land plant evolution. The evidence presented here strongly suggests that plant-mycorrhizal fungus symbiosis was one of the key processes that contributed to the origin of land flora.