SETD5 gene mutations have been identified as a frequent cause of idiopathic intellectual disability. Here we show that Setd5-haploinsufficient mice present developmental defects such as abnormal brain-to-body weight ratios and neural crest defect-associated phenotypes. Furthermore, Setd5-mutant mice show impairments in cognitive tasks, enhanced long-term potentiation, delayed ontogenetic profile of ultrasonic vocalization, and behavioral inflexibility. Behavioral issues are accompanied by abnormal expression of postsynaptic density proteins previously associated with cognition. Our data additionally indicate that Setd5 regulates RNA polymerase II dynamics and gene transcription via its interaction with the Hdac3 and Paf1 complexes, findings potentially explaining the gene expression defects observed in Setd5-haploinsufficient mice. Our results emphasize the decisive role of Setd5 in a biological pathway found to be disrupted in humans with intellectual disability and autism spectrum disorder.