It is well known that Cu diffuses faster than Ag in covalent semiconductors such as Si, which has prevented the replacement of Ag by Cu as a contact material in Si solar cells for reducing the cost. Surprisingly, in more ionic materials such as CdTe, Ag diffuses faster than Cu despite that it is larger than Cu, which has prevented the replacement of Cu by Ag in CdTe solar cells to improve the performance. But, so far, the mechanisms behind these distinct diffusion behaviors of Cu and Ag in covalent and ionic semiconductors have not been addressed. Here we reveal the underlying mechanisms by combining the first-principles calculations and group theory analysis. We find that the symmetry controlled s-d coupling plays a critical role in determining the diffusion behaviors. The s-d coupling is absent in pure covalent semiconductors but increases with the ionicity of the zinc blende semiconductors, and is larger for Cu than for Ag, owing to its higher d orbital energy. In conjunction with Coulomb interaction and strain energy, the s-d coupling is able to explain all the diffusion behaviors from Cu to Ag and from covalent to ionic hosts. This in-depth understanding enables us to engineer the diffusion of impurities in various semiconductors.