Serotonin, behavior, and natural selection in New World monkeys

Abstract

Traits that undergo massive natural selection pressure, with multiple events of positive selection, are hard to find. Social behaviour, in social animals, is crucial for survival, and genetic networks involved in behaviour, such as those of serotonin (5-HT) and other neurotransmitters, must be the target of natural selection. Here, we used molecular analyses to search for signals of positive selection in the 5-HT system and found such signals in the M3-M4 intracellular domain of the 5-HT3A serotonin receptor subunit (HTR3A) in primates. We detected four amino acid sites with signs of putatively positive selection (398, 403, 432 and 416); the first three showed indications of being selected in New World monkeys (NWM, Platyrrhini), specifically in the Callitrichinae branch. Additionally, we searched for associations of these amino acid variants with social behavioural traits (i.e. sex-biased dispersal, dominance and social monogamy) using classical and Bayesian methods, and found statistically significant associations for unbiased sex dispersal (398L and 416S), unbiased sex dominance (416S) and social monogamy (416S), as well as significant positive correlation between female dispersal and 403G. Furthermore, we found putatively functional protein motifs determined by three selected sites, of which we highlight a ligand motif to GSK3 in the 416S variant, appearing only in Platyrrhini. 5-HT, 5-HT3A receptor and GSK3 are part of a network that participates in neurodevelopment and regulates behaviour, among other functions. We suggest that these genetic variations, together with those found in other neurotransmitter systems, must contribute to adaptive behaviours and consequently to fitness in NWMs.

Keywords: 5-HT3 receptor; Platyrrhini; mating system; neurotransmission.