G'ST and D do not replace FST

Mol Ecol. 2011 Mar;20(6):1083-91. doi: 10.1111/j.1365-294X.2010.04996.x. Epub 2011 Jan 19.

Abstract

The genetic differentiation among populations is affected by mutation as well as by migration, drift and selection. For loci with high mutation rates, such as microsatellites, the amount of mutation can influence the values of indices of differentiation such as G(ST) and F(ST). For many purposes, this effect is undesirable, and as a result, new indices such as G'(ST) and D have been proposed to measure population differentiation. This paper shows that these new indices are not effective measures of the causes or consequences of population structure. Both G'(ST) and D depend heavily on mutation rate, but both are insensitive to any population genetic process when the mutation rate is high relative to the migration rate. Furthermore, D is specific to the locus being measured, and so little can be inferred about the population demography from D. However, at equilibrium, D may provide an index of whether a particular marker is more strongly affected by mutation than by migration. I argue that F(ST) is a more important summary of the effects of population structure than D and that R(ST) or other measures that explicitly account for the mutation process are much better than G(ST), G'(ST), or D for highly mutable markers. Markers with lower mutation rates will often be easier to interpret.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Genetic Drift
  • Genetics, Population / methods*
  • Models, Theoretical*
  • Population Dynamics