Genome-wide association studies on bipolar disorders (BD) have revealed an additive polygenic contribution of common single-nucleotide polymorphisms (SNPs). However, these SNPs explain only 25% of the overall genetic variance and suggest a role of rare variants in BD vulnerability. Here, we combined high-throughput genotyping data and whole-exome sequencing in cohorts of individuals with BD as well as in multiplex families with a high density of affected individuals in order to determine the contribution of both common and rare variants to BD genetic vulnerability. Using polygenic risk scores (PRS), we showed a strong contribution of common polymorphisms previously associated with BD and schizophrenia (SZ) and noticed that those specifically associated with SZ contributed more in familial forms of BD than in non-familial ones. The analysis of rare damaging variants shared by affected individuals in multiplex families with BD revealed a single interaction network enriched in neuronal and developmental biological pathways, as well as in the regulation of gene expression. We identified four genes with a higher mutation rate in individuals with BD than in the general population and showed that mutations in two of them were associated with specific clinical manifestations. In addition, we showed a significant negative correlation between PRS and the number of rare damaging variants specifically in unaffected individuals of multiplex families. Altogether, our results suggest that common and rare genetic variants both contribute to the familial aggregation of BD and this genetic architecture may explain the heterogeneity of clinical manifestations in multiplex families.