Dup(17)(p11.2) syndrome, consisting of a spectrum of more than 20 clinical features, is associated with increased dosage of the retinoic acid induced 1 (RAI1) gene. We previously reported on the generation and evaluation of Rai1-overexpressing mice. Several phenotypes, including increased anxiety and hyperactivity, growth retardation, and altered motor and sensory coordination, were observed, recapitulating phenotypes observed in patients with 17p11.2 duplication. In addition, these mice have reduced reproductive fitness. In this study we expand investigations to identify possible neural correlates for increased Rai1 dosage. We analyzed Rai1-transgenic breeding data and evaluated maternal and social behaviors as potential causes for reduced litter size in Rai1 transgenics compared to wild-type controls. Abnormal maternal behavior, including delayed pup retrieval in the Rai1-transgenic dams compared to wild-type dams, was identified. Mendelian transmission of parental genotypes was also distorted in the pups from transgenic breeding. Furthermore, altered social behavior was observed in the male transgenic mice. Analysis of neurotransmitter levels from whole-brain lysates showed significantly impaired serotonin metabolism indicating a neuronal basis for behavioral modifications in these mice. Our study suggests an important role for Rai1 in the serotonin pathway in a dosage-dependent manner.