The Carlemanniaceae comprises two small genera that are restricted to East Asia: the Carlemannia and Silvianthus. These genera were previously placed in the Rubiaceae or Caprifoliaceae, but are now considered a distinct family that is probably related to the Oleaceae in the Lamiales. The family is still poorly understood with respect to its morphological characteristics. Here, we present the first report of the chromosome numbers of the family using species from both genera, i.e., Carlemannia tetragona, Silvianthus bracteatus ssp. bracteatus, and S. bracteatus ssp. clerodendroides. The species were compared with the chromosome numbers of Oleaceae and associated families using a Bayesian tree that was generated from rbcL and ndhF sequence data from Genbank. C. tetragona had 2n = 30 (x = 15), whereas the two subspecies of Silvianthus had 2n = 38 (x = 19). Comparisons of chromosome numbers support the distinctness of the Carlemanniaceae, not only from the Oleaceae (x = 11, 13, 23), but also from the Tetrachondraceae (x = 10, 11), a family that is possibly related to the Carlemanniaceae and/or Oleaceae in the Lamiales. The notable difference in chromosome number between Carlemannia and Silvianthus, as well as the differences in other characteristics (pollen, seed, and fruit morphology), suggests that the family split early in its evolution.